[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:03 dist-test-slave-zx46 krb5kdc[15877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:03 dist-test-slave-zx46 krb5kdc[15877](info): set up 2 sockets
Sep 24 07:16:03 dist-test-slave-zx46 krb5kdc[15877](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 07:16:04.693547 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:04 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:04.819764 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33113
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:04.930116 15890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:04.930373 15890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:04.930411 15890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:04.934325 15890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:04.934376 15890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:04.934396 15890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:04.934412 15890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:04.934428 15890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:04.939558 15890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:04.940565 15890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:04.941570 15890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:04.951486 15898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:04.951480 15895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:04.951480 15896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:04.952078 15890 server_base.cc:1034] running on GCE node
I20260924 07:16:04.952862 15890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:04.954368 15890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:04.955575 15890 hybrid_clock.cc:648] HybridClock initialized: now 1790234164955485 us; error 110 us; skew 500 ppm
Sep 24 07:16:04 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234164, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:04.958549 15890 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:04.959421 15890 webserver.cc:460] Webserver started at http://127.15.127.190:46195/ using document root <none> and password file <none>
I20260924 07:16:04.959980 15890 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:04.960067 15890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:04.960300 15890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:04.962407 15890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "3cda8966968a4f34a5e3ca348302e04e"
format_stamp: "Formatted at 2026-09-24 07:16:04 on dist-test-slave-zx46"
server_key: "6b9ca23568b3a8246651778124fd72ec"
server_key_iv: "be4f27dc98c82e0b0fa361a4a8fd6841"
server_key_version: "encryptionkey@0"
I20260924 07:16:04.963004 15890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "3cda8966968a4f34a5e3ca348302e04e"
format_stamp: "Formatted at 2026-09-24 07:16:04 on dist-test-slave-zx46"
server_key: "6b9ca23568b3a8246651778124fd72ec"
server_key_iv: "be4f27dc98c82e0b0fa361a4a8fd6841"
server_key_version: "encryptionkey@0"
I20260924 07:16:04.966745 15890 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:16:04.969141 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:04.970374 15890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:04.970521 15890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "3cda8966968a4f34a5e3ca348302e04e"
format_stamp: "Formatted at 2026-09-24 07:16:04 on dist-test-slave-zx46"
server_key: "6b9ca23568b3a8246651778124fd72ec"
server_key_iv: "be4f27dc98c82e0b0fa361a4a8fd6841"
server_key_version: "encryptionkey@0"
I20260924 07:16:04.970631 15890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:04.989545 15890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:04.992437 15890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:04.992640 15890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:04.999657 15890 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33113
I20260924 07:16:04.999676 15957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33113 every 8 connection(s)
I20260924 07:16:05.000823 15890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:05.003646 15958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:05.005991 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 15890
I20260924 07:16:05.006110 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:05.007402 15870 external_mini_cluster.cc:1428] Setting key 41b6881f4299820e4c7b5dab0ed758c6
I20260924 07:16:05.009222 15958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e: Bootstrap starting.
I20260924 07:16:05.012003 15958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:05.012710 15958 log.cc:824] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:05.014492 15958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e: No bootstrap required, opened a new log
I20260924 07:16:05.018003 15958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cda8966968a4f34a5e3ca348302e04e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33113 } }
I20260924 07:16:05.018230 15958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:05.018317 15958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cda8966968a4f34a5e3ca348302e04e, State: Initialized, Role: FOLLOWER
I20260924 07:16:05.018771 15958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cda8966968a4f34a5e3ca348302e04e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33113 } }
I20260924 07:16:05.018920 15958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:05.019007 15958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:05.019102 15958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:05.019953 15958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cda8966968a4f34a5e3ca348302e04e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33113 } }
I20260924 07:16:05.020275 15958 leader_election.cc:302] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cda8966968a4f34a5e3ca348302e04e; no voters:
I20260924 07:16:05.020582 15958 leader_election.cc:288] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:05.020648 15963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:05.020875 15963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [term 1 LEADER]: Becoming Leader. State: Replica: 3cda8966968a4f34a5e3ca348302e04e, State: Running, Role: LEADER
I20260924 07:16:05.021229 15963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cda8966968a4f34a5e3ca348302e04e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33113 } }
I20260924 07:16:05.021816 15958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:05.022876 15964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cda8966968a4f34a5e3ca348302e04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cda8966968a4f34a5e3ca348302e04e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33113 } } }
I20260924 07:16:05.023010 15964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:05.023301 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cda8966968a4f34a5e3ca348302e04e. Latest consensus state: current_term: 1 leader_uuid: "3cda8966968a4f34a5e3ca348302e04e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cda8966968a4f34a5e3ca348302e04e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33113 } } }
I20260924 07:16:05.023417 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:05.024302 15972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:05.027547 15972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:05.031913 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.010384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37178 (local address 127.15.127.190:33113)
0924 07:16:05.010772 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:05.010787 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:05.011881 (+ 1094us) server_negotiation.cc:407] Connection header received
0924 07:16:05.015090 (+ 3209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:05.015111 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:05.015422 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:05.015732 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:05.017879 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.019046 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.020275 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.021052 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.021235 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.025992 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:05.025998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:05.026013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:05.026048 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:05.028203 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.028886 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.028907 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.028913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.029080 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.029495 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.029498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.029500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.029983 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:05.030179 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:05.030830 (+ 651us) server_negotiation.cc:299] Negotiation successful
0924 07:16:05.031111 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":121,"threads_started":1}
I20260924 07:16:05.035004 15972 catalog_manager.cc:1347] Generated new cluster ID: 2a675e4810b344ba936157d0e60fe74e
I20260924 07:16:05.035082 15972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:05.054471 15972 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:05.055469 15972 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:05.063969 15972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cda8966968a4f34a5e3ca348302e04e: Generated new TSK 0
I20260924 07:16:05.064664 15972 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:05.099858 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33113
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:05.210105 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:05.210357 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:05.210399 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:05.214259 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:05.214308 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:05.214363 15984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:05.219515 15984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:05.220441 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:05.221441 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:05.231197 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:05.231206 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:05.231254 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:05.231246 15984 server_base.cc:1034] running on GCE node
I20260924 07:16:05.231988 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:05.232584 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:05.233747 15984 hybrid_clock.cc:648] HybridClock initialized: now 1790234165233733 us; error 25 us; skew 500 ppm
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:05.236740 15984 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:05.237691 15984 webserver.cc:460] Webserver started at http://127.15.127.129:46475/ using document root <none> and password file <none>
I20260924 07:16:05.238257 15984 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:05.238348 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:05.238592 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:05.240710 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "d07eca673aae4f389216eddfd5469e1c"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "c28ab17de2f1c837a054e1e1b9cd4ff0"
server_key_iv: "47ecd82edc1bfd35d750f883955cac72"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.241364 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "d07eca673aae4f389216eddfd5469e1c"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "c28ab17de2f1c837a054e1e1b9cd4ff0"
server_key_iv: "47ecd82edc1bfd35d750f883955cac72"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.245484 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:16:05.247879 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.249181 15984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:05.249318 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "d07eca673aae4f389216eddfd5469e1c"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "c28ab17de2f1c837a054e1e1b9cd4ff0"
server_key_iv: "47ecd82edc1bfd35d750f883955cac72"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.249435 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:05.270893 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:05.274492 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:05.274695 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:05.275344 15984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:05.276194 15984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:05.276260 15984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.276326 15984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:05.276377 15984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.285331 15984 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:44853
I20260924 07:16:05.285354 16112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:44853 every 8 connection(s)
I20260924 07:16:05.286285 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:05.296061 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 15984
I20260924 07:16:05.296172 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:05.296814 15870 external_mini_cluster.cc:1428] Setting key e8a09b57c8dbe21d8a7ecbcb93e765da
I20260924 07:16:05.299779 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.288044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:36195 (local address 127.15.127.190:33113)
0924 07:16:05.288206 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:05.288211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:05.289463 (+ 1252us) server_negotiation.cc:407] Connection header received
0924 07:16:05.290313 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:05.290316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:05.290361 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:05.290434 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:05.291598 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.292137 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.293284 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.293512 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.293591 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.295904 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:05.295910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:05.295916 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:05.295948 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:05.297634 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.298146 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.298151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.298153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.298242 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.298658 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.298660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.298661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.298831 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:05.298905 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:05.299558 (+ 653us) server_negotiation.cc:299] Negotiation successful
0924 07:16:05.299665 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:16:05.300700 16115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.288449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33113 (local address 127.15.127.129:36195)
0924 07:16:05.289320 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:05.289364 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:05.290118 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:05.290614 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:05.290621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:05.290986 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:05.291465 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:05.291479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.292238 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.292240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:05.293183 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:05.293192 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.293636 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.293639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:05.293771 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.294317 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:05.294333 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:05.295730 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:05.297749 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:05.297754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:05.297765 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:05.298033 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:05.298345 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:05.298365 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:05.298367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:05.298554 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:05.298942 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:05.298951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:05.299257 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 07:16:05.299475 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 07:16:05.299819 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":108,"threads_started":1}
I20260924 07:16:05.302022 16113 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33113
I20260924 07:16:05.302271 16113 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:05.302799 16113 heartbeater.cc:511] Master 127.15.127.190:33113 requested a full tablet report, sending...
I20260924 07:16:05.304226 15922 ts_manager.cc:194] Registered new tserver with Master: d07eca673aae4f389216eddfd5469e1c (127.15.127.129:44853)
I20260924 07:16:05.305449 15922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:36195
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:05.323906 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33113
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:05.433405 16118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:05.433673 16118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:05.433753 16118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:05.437704 16118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:05.437788 16118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:05.437891 16118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:05.444468 16118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:05.445535 16118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:05.446548 16118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:05.456262 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:05.456390 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:05.456283 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:05.456609 16118 server_base.cc:1034] running on GCE node
I20260924 07:16:05.456982 16118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:05.457679 16118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:05.458839 16118 hybrid_clock.cc:648] HybridClock initialized: now 1790234165458810 us; error 36 us; skew 500 ppm
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:05.461480 16118 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:05.462234 16118 webserver.cc:460] Webserver started at http://127.15.127.130:37547/ using document root <none> and password file <none>
I20260924 07:16:05.462743 16118 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:05.462798 16118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:05.462955 16118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:05.464884 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "590de30c0d58410e94539d9d937b0ada"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "11175c41962e6a6fc2a9e492d191e0d5"
server_key_iv: "11f5d82b8adb3e86e375fb440edd0eee"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.465462 16118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "590de30c0d58410e94539d9d937b0ada"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "11175c41962e6a6fc2a9e492d191e0d5"
server_key_iv: "11f5d82b8adb3e86e375fb440edd0eee"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.468979 16118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:05.471251 16133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.472430 16118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:05.472518 16118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "590de30c0d58410e94539d9d937b0ada"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "11175c41962e6a6fc2a9e492d191e0d5"
server_key_iv: "11f5d82b8adb3e86e375fb440edd0eee"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.472604 16118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:05.494470 16118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:05.497780 16118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:05.497928 16118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:05.498447 16118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:05.499207 16118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:05.499259 16118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.499297 16118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:05.499315 16118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.507841 16118 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:37975
I20260924 07:16:05.507912 16246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:37975 every 8 connection(s)
I20260924 07:16:05.508745 16118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:05.509995 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16118
I20260924 07:16:05.510109 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:05.510416 15870 external_mini_cluster.cc:1428] Setting key 3b3d766bbc044045e883ceb8fbbbcaff
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:05.523026 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.510654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:51761 (local address 127.15.127.190:33113)
0924 07:16:05.510810 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:05.510815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:05.512265 (+ 1450us) server_negotiation.cc:407] Connection header received
0924 07:16:05.513179 (+ 914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:05.513182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:05.513235 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:05.513312 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:05.514483 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.515009 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.516153 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.516379 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.516490 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.518820 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:05.518823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:05.518825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:05.518847 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:05.520346 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.520905 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.520909 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.520911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.520992 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.521452 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.521455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.521456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.521628 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:05.521730 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:05.522646 (+ 916us) server_negotiation.cc:299] Negotiation successful
0924 07:16:05.522784 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:16:05.524022 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.511098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33113 (local address 127.15.127.130:51761)
0924 07:16:05.512099 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:05.512141 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:05.512937 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:05.513472 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:05.513478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:05.513893 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:05.514345 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:05.514358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.515127 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.515131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:05.515984 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:05.515992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.516523 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.516527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:05.516700 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.517295 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:05.517311 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:05.518622 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:05.520478 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:05.520484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:05.520498 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:05.520801 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:05.521155 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:05.521158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:05.521160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:05.521290 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:05.521890 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:05.521895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:05.522265 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 07:16:05.522581 (+ 316us) client_negotiation.cc:240] Negotiation successful
0924 07:16:05.522883 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":642,"threads_started":1}
I20260924 07:16:05.525334 16247 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33113
I20260924 07:16:05.525573 16247 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:05.526041 16247 heartbeater.cc:511] Master 127.15.127.190:33113 requested a full tablet report, sending...
I20260924 07:16:05.527172 15922 ts_manager.cc:194] Registered new tserver with Master: 590de30c0d58410e94539d9d937b0ada (127.15.127.130:37975)
I20260924 07:16:05.528062 15922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:51761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:05.540043 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33113
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:05.650806 16252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:05.651088 16252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:05.651176 16252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:05.655690 16252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:05.655769 16252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:05.655860 16252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:05.661149 16252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33113
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:05.662145 16252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:05.663110 16252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:05.672673 16260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:05.672788 16257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:05.672859 16252 server_base.cc:1034] running on GCE node
W20260924 07:16:05.672670 16258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:05.673395 16252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:05.674073 16252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:05.675241 16252 hybrid_clock.cc:648] HybridClock initialized: now 1790234165675219 us; error 34 us; skew 500 ppm
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:05.678047 16252 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:05.678948 16252 webserver.cc:460] Webserver started at http://127.15.127.131:34941/ using document root <none> and password file <none>
I20260924 07:16:05.679520 16252 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:05.679608 16252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:05.679839 16252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:05.681917 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "981a9681caa847d6a7ae291af3b0149f"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "c770eb292c1c52d074b8f4f9d27b2479"
server_key_iv: "3442f4c4867d36f7155fc5e479055106"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.682473 16252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "981a9681caa847d6a7ae291af3b0149f"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "c770eb292c1c52d074b8f4f9d27b2479"
server_key_iv: "3442f4c4867d36f7155fc5e479055106"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.686105 16252 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:05.688470 16267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.689779 16252 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:05.689931 16252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "981a9681caa847d6a7ae291af3b0149f"
format_stamp: "Formatted at 2026-09-24 07:16:05 on dist-test-slave-zx46"
server_key: "c770eb292c1c52d074b8f4f9d27b2479"
server_key_iv: "3442f4c4867d36f7155fc5e479055106"
server_key_version: "encryptionkey@0"
I20260924 07:16:05.690054 16252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:05.709510 16252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:05.712432 16252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:05.712628 16252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:05.713297 16252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:05.714102 16252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:05.714170 16252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.714238 16252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:05.714288 16252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:05.723734 16252 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:38111
I20260924 07:16:05.723752 16380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:38111 every 8 connection(s)
I20260924 07:16:05.724673 16252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:05.725816 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16252
I20260924 07:16:05.725943 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:05.726212 15870 external_mini_cluster.cc:1428] Setting key ed5ac103063678fa5e92ded3f8510e53
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:05.737783 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.726538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:39679 (local address 127.15.127.190:33113)
0924 07:16:05.726702 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:05.726707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:05.727618 (+ 911us) server_negotiation.cc:407] Connection header received
0924 07:16:05.729125 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:05.729128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:05.729173 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:05.729240 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:05.730506 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.731026 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.732112 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.732350 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.732426 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.734766 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:05.734770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:05.734772 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:05.734796 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:05.736000 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.736455 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.736458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.736459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.736529 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.736818 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.736820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.736821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.736985 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:05.737117 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:05.737564 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 07:16:05.737661 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:16:05.738610 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.727023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33113 (local address 127.15.127.131:39679)
0924 07:16:05.727480 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:05.727531 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:05.728874 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:05.729403 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:05.729410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:05.729841 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:05.730378 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:05.730391 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.731126 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.731129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:05.732004 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:05.732012 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.732473 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.732475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:05.732607 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.733154 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:05.733171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:05.734609 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:05.736099 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:05.736104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:05.736116 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:05.736370 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:05.736607 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:05.736610 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:05.736611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:05.736743 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:05.737122 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:05.737127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:05.737362 (+ 235us) client_negotiation.cc:769] Sending connection context
0924 07:16:05.737517 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 07:16:05.737759 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":129,"threads_started":1}
I20260924 07:16:05.739766 16381 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33113
I20260924 07:16:05.740039 16381 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:05.740809 16381 heartbeater.cc:511] Master 127.15.127.190:33113 requested a full tablet report, sending...
I20260924 07:16:05.741804 15922 ts_manager.cc:194] Registered new tserver with Master: 981a9681caa847d6a7ae291af3b0149f (127.15.127.131:38111)
I20260924 07:16:05.742321 15922 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:39679
I20260924 07:16:05.752393 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:05.778406 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.768987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.15.127.129:44853)
0924 07:16:05.769352 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:05.769366 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:05.769391 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:16:05.769497 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:05.769502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:05.769697 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:05.769817 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:05.770447 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.771230 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.772037 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.772410 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.772495 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.774916 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:05.774925 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:05.774938 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:05.774960 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:05.776858 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.777284 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.777290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.777291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.777381 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.777592 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.777594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.777595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.777820 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:05.777970 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:05.778118 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:16:05.778240 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":102,"threads_started":1}
W20260924 07:16:05.779428 16027 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48120
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[15877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:05.804461 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:05.796453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.15.127.129:44853)
0924 07:16:05.796623 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:05.796628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:05.796754 (+ 126us) server_negotiation.cc:407] Connection header received
0924 07:16:05.796936 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:05.796940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:05.796991 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:05.797100 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:05.797782 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.798310 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.799174 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:05.799426 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:05.799535 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:05.801891 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:05.801894 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:05.801896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:05.801919 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:05.803127 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.803527 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.803529 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.803530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.803600 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:05.803810 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:05.803812 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:05.803813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:05.803968 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:05.804064 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:05.804242 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:16:05.804340 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:16:05.805603 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 15984
I20260924 07:16:05.812215 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16118
I20260924 07:16:05.819227 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16252
I20260924 07:16:05.824685 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 15890
2026-09-24T07:16:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2177 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[16395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[16395](info): set up 2 sockets
Sep 24 07:16:05 dist-test-slave-zx46 krb5kdc[16395](info): commencing operation
krb5kdc: starting...
W20260924 07:16:06.889526 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 07:16:07.023979 15870 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.134s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 07:16:07.161882 15870 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.138s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 07:16:07.297114 15870 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.135s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/test-user.keytab.
W20260924 07:16:08.408023 15870 mini_kdc.cc:302] Time spent creating keytab for test-user: real 1.111s user 0.000s sys 0.002s
Sep 24 07:16:08 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
W20260924 07:16:09.426806 15870 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.15.127.190: real 0.995s user 0.000s sys 0.003s
I20260924 07:16:09.427369 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:39917
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:39917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:09.547811 16408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:09.548108 16408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:09.548151 16408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:09.552263 16408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:09.552335 16408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:09.552356 16408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:09.552371 16408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:09.552387 16408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:09.558197 16408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:39917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:09.559319 16408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:09.560364 16408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:09.569347 16414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:09.569332 16413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:09.569504 16408 server_base.cc:1034] running on GCE node
W20260924 07:16:09.569332 16416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:09.570256 16408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:09.571451 16408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:09.572614 16408 hybrid_clock.cc:648] HybridClock initialized: now 1790234169572585 us; error 47 us; skew 500 ppm
Sep 24 07:16:09 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234169, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:09.581897 16408 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:09.582998 16408 webserver.cc:460] Webserver started at http://127.15.127.190:39925/ using document root <none> and password file <none>
I20260924 07:16:09.583626 16408 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:09.583729 16408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:09.583969 16408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:09.586215 16408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "26edffc82d8543678fda33abd90a2c99"
format_stamp: "Formatted at 2026-09-24 07:16:09 on dist-test-slave-zx46"
server_key: "3b2e5443a9a27c8bebd21ee28458d8d1"
server_key_iv: "ae128b0ad3a0d2d4864bb833ae37e255"
server_key_version: "encryptionkey@0"
I20260924 07:16:09.586802 16408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "26edffc82d8543678fda33abd90a2c99"
format_stamp: "Formatted at 2026-09-24 07:16:09 on dist-test-slave-zx46"
server_key: "3b2e5443a9a27c8bebd21ee28458d8d1"
server_key_iv: "ae128b0ad3a0d2d4864bb833ae37e255"
server_key_version: "encryptionkey@0"
I20260924 07:16:09.590747 16408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:09.593247 16423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:09.594506 16408 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:09.594645 16408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "26edffc82d8543678fda33abd90a2c99"
format_stamp: "Formatted at 2026-09-24 07:16:09 on dist-test-slave-zx46"
server_key: "3b2e5443a9a27c8bebd21ee28458d8d1"
server_key_iv: "ae128b0ad3a0d2d4864bb833ae37e255"
server_key_version: "encryptionkey@0"
I20260924 07:16:09.594763 16408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:09.622519 16408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:09.625965 16408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:09.626175 16408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:09.634207 16408 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:39917
I20260924 07:16:09.634234 16475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:39917 every 8 connection(s)
I20260924 07:16:09.635182 16408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:09.638268 16476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:09.644698 16476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99: Bootstrap starting.
I20260924 07:16:09.644901 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16408
I20260924 07:16:09.645042 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:09.645308 15870 external_mini_cluster.cc:1428] Setting key 11047e69838856a1c1f834c8ae72f2fb
I20260924 07:16:09.647586 16476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:09.648353 16476 log.cc:824] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:09.650380 16476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99: No bootstrap required, opened a new log
Sep 24 07:16:09 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:09.654299 16476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26edffc82d8543678fda33abd90a2c99" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39917 } }
I20260924 07:16:09.654575 16476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:09.654639 16476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26edffc82d8543678fda33abd90a2c99, State: Initialized, Role: FOLLOWER
I20260924 07:16:09.655179 16476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26edffc82d8543678fda33abd90a2c99" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39917 } }
I20260924 07:16:09.655309 16476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:09.655395 16476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:09.655495 16476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:09.659736 16479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:09.646783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.15.127.190:39917)
0924 07:16:09.647223 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:09.647239 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:09.647280 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:16:09.648029 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:09.648049 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:09.648421 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:09.648835 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:09.649637 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:09.650686 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:09.651561 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:09.652042 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:09.652207 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:09.655133 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:09.655138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:09.655153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:09.655180 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:09.656978 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:09.657451 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:09.657456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:09.657461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:09.657574 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:09.657815 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:09.657817 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:09.657819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:09.658218 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:09.658373 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:09.658591 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 07:16:09.658930 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260924 07:16:09.719589 16476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26edffc82d8543678fda33abd90a2c99" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39917 } }
I20260924 07:16:09.720122 16476 leader_election.cc:302] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26edffc82d8543678fda33abd90a2c99; no voters:
I20260924 07:16:09.720536 16476 leader_election.cc:288] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:09.720697 16481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:09.720943 16481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [term 1 LEADER]: Becoming Leader. State: Replica: 26edffc82d8543678fda33abd90a2c99, State: Running, Role: LEADER
I20260924 07:16:09.721431 16481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26edffc82d8543678fda33abd90a2c99" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39917 } }
I20260924 07:16:09.722059 16476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:09.723194 16483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26edffc82d8543678fda33abd90a2c99. Latest consensus state: current_term: 1 leader_uuid: "26edffc82d8543678fda33abd90a2c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26edffc82d8543678fda33abd90a2c99" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39917 } } }
I20260924 07:16:09.723337 16483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:09.723690 16482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26edffc82d8543678fda33abd90a2c99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26edffc82d8543678fda33abd90a2c99" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39917 } } }
I20260924 07:16:09.723780 16482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:09.724584 16490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:09.727696 16490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:09.733739 16490 catalog_manager.cc:1347] Generated new cluster ID: 1559292c47974f51b146ee22e8815632
I20260924 07:16:09.733819 16490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:09.743192 16490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:09.744268 16490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:09.752120 16490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26edffc82d8543678fda33abd90a2c99: Generated new TSK 0
I20260924 07:16:09.752978 16490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
W20260924 07:16:10.777212 15870 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.15.127.129: real 1.012s user 0.002s sys 0.001s
I20260924 07:16:10.777676 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39917
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:10.886996 16502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:10.887272 16502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:10.887333 16502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:10.891283 16502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:10.891340 16502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:10.891399 16502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:10.896711 16502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39917
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:10.897797 16502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:10.898813 16502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:10.908614 16507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:10.908746 16502 server_base.cc:1034] running on GCE node
W20260924 07:16:10.908614 16508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:10.908944 16510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:10.909425 16502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:10.910040 16502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:10.911216 16502 hybrid_clock.cc:648] HybridClock initialized: now 1790234170911184 us; error 47 us; skew 500 ppm
Sep 24 07:16:10 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234170, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:10.914044 16502 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:10.914888 16502 webserver.cc:460] Webserver started at http://127.15.127.129:46495/ using document root <none> and password file <none>
I20260924 07:16:10.915433 16502 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:10.915490 16502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:10.915656 16502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:10.917788 16502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "4fb6620dfd504313a5548831e123379f"
format_stamp: "Formatted at 2026-09-24 07:16:10 on dist-test-slave-zx46"
server_key: "f96d51bc964b9be7240cc8b630ad3422"
server_key_iv: "d2ce6dfe2b6eaa89e3baf35ea1bcf769"
server_key_version: "encryptionkey@0"
I20260924 07:16:10.918303 16502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "4fb6620dfd504313a5548831e123379f"
format_stamp: "Formatted at 2026-09-24 07:16:10 on dist-test-slave-zx46"
server_key: "f96d51bc964b9be7240cc8b630ad3422"
server_key_iv: "d2ce6dfe2b6eaa89e3baf35ea1bcf769"
server_key_version: "encryptionkey@0"
I20260924 07:16:10.922051 16502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:16:10.924649 16517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:10.925962 16502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:10.926059 16502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "4fb6620dfd504313a5548831e123379f"
format_stamp: "Formatted at 2026-09-24 07:16:10 on dist-test-slave-zx46"
server_key: "f96d51bc964b9be7240cc8b630ad3422"
server_key_iv: "d2ce6dfe2b6eaa89e3baf35ea1bcf769"
server_key_version: "encryptionkey@0"
I20260924 07:16:10.926148 16502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:10.938086 16502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:10.941334 16502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:10.941488 16502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:10.942129 16502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:10.942989 16502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:10.943037 16502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:10.943102 16502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:10.943133 16502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:10.951606 16502 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:43733
I20260924 07:16:10.951653 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:43733 every 8 connection(s)
I20260924 07:16:10.952545 16502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:10.953712 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16502
I20260924 07:16:10.953822 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:10.954099 15870 external_mini_cluster.cc:1428] Setting key d3477b96bc61b1cd0e26e29c1a871e08
Sep 24 07:16:10 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234170, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:16:10.967432 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:10.954615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:40505 (local address 127.15.127.190:39917)
0924 07:16:10.954899 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:10.954903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:10.955899 (+ 996us) server_negotiation.cc:407] Connection header received
0924 07:16:10.956993 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:10.956996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:10.957096 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:10.957202 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:10.958436 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:10.958991 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:10.960170 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:10.960412 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:10.960519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:10.963293 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:10.963296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:10.963298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:10.963325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:10.964977 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:10.965575 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:10.965577 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:10.965579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:10.965660 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:10.966091 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:10.966097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:10.966101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:10.966363 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:10.966470 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:10.966972 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 07:16:10.967126 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":116,"threads_started":1}
I20260924 07:16:10.968394 16634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:10.955124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:39917 (local address 127.15.127.129:40505)
0924 07:16:10.955711 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:10.955781 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:10.956772 (+ 991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:10.957400 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:10.957412 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:10.957785 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:10.958285 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:10.958299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:10.959118 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:10.959122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:10.960033 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:10.960042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:10.960563 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:10.960567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:10.960749 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:10.961397 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:10.961414 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:10.963089 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:10.965142 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:10.965147 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:10.965161 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:10.965450 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:10.965747 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:10.965752 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:10.965757 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:10.965969 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:10.966469 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:10.966475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:10.966725 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 07:16:10.966892 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 07:16:10.967161 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":148,"threads_started":1}
I20260924 07:16:10.969726 16631 heartbeater.cc:347] Connected to a master server at 127.15.127.190:39917
I20260924 07:16:10.970021 16631 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:10.970659 16631 heartbeater.cc:511] Master 127.15.127.190:39917 requested a full tablet report, sending...
I20260924 07:16:10.972330 16440 ts_manager.cc:194] Registered new tserver with Master: 4fb6620dfd504313a5548831e123379f (127.15.127.129:43733)
I20260924 07:16:10.973541 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:40505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:10.983886 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39917
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:11.093273 16637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:11.093575 16637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:11.093650 16637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:11.097738 16637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:11.097833 16637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:11.097932 16637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:11.103338 16637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39917
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:11.104432 16637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:11.105489 16637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:11.114830 16642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:11.114830 16643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:11.114867 16645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:11.115219 16637 server_base.cc:1034] running on GCE node
I20260924 07:16:11.115631 16637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:11.116288 16637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:11.117442 16637 hybrid_clock.cc:648] HybridClock initialized: now 1790234171117425 us; error 30 us; skew 500 ppm
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:11.120230 16637 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:11.121191 16637 webserver.cc:460] Webserver started at http://127.15.127.130:37861/ using document root <none> and password file <none>
I20260924 07:16:11.121807 16637 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:11.121899 16637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:11.122129 16637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:11.124400 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "98c41f56f12c43f8939289802cd55055"
format_stamp: "Formatted at 2026-09-24 07:16:11 on dist-test-slave-zx46"
server_key: "f2cdb1c12fd196e1b88ce335af850a08"
server_key_iv: "aaa293d55db5153adccf4301acef9696"
server_key_version: "encryptionkey@0"
I20260924 07:16:11.124962 16637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "98c41f56f12c43f8939289802cd55055"
format_stamp: "Formatted at 2026-09-24 07:16:11 on dist-test-slave-zx46"
server_key: "f2cdb1c12fd196e1b88ce335af850a08"
server_key_iv: "aaa293d55db5153adccf4301acef9696"
server_key_version: "encryptionkey@0"
I20260924 07:16:11.128574 16637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:16:11.131155 16652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:11.132401 16637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:11.132550 16637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "98c41f56f12c43f8939289802cd55055"
format_stamp: "Formatted at 2026-09-24 07:16:11 on dist-test-slave-zx46"
server_key: "f2cdb1c12fd196e1b88ce335af850a08"
server_key_iv: "aaa293d55db5153adccf4301acef9696"
server_key_version: "encryptionkey@0"
I20260924 07:16:11.132663 16637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:11.168093 16637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:11.171857 16637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:11.172072 16637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:11.172698 16637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:11.173576 16637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:11.173648 16637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:11.173712 16637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:11.173756 16637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:11.182400 16637 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:34617
I20260924 07:16:11.182418 16765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:34617 every 8 connection(s)
I20260924 07:16:11.183321 16637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:11.190670 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16637
I20260924 07:16:11.190768 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:11.191021 15870 external_mini_cluster.cc:1428] Setting key d8e79beb05fbbccb92a6c91f85af2022
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:11.197253 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.185125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:37461 (local address 127.15.127.190:39917)
0924 07:16:11.185253 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.185256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.186138 (+ 882us) server_negotiation.cc:407] Connection header received
0924 07:16:11.187041 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.187045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.187141 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.187242 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:11.188393 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.188924 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.190139 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.190376 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.190505 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.193227 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:11.193234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:11.193240 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:11.193270 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:11.194832 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.195462 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.195469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.195473 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.195566 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.195918 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.195924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.195928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.196195 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:11.196326 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.196816 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.196976 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:16:11.197984 16768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.185513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:39917 (local address 127.15.127.130:37461)
0924 07:16:11.185960 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.186002 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.186820 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.187436 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.187443 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.187789 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:11.188253 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.188266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.189075 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.189079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.189973 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.189981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.190545 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.190548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.190696 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.191423 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:11.191443 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:11.192987 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:11.194955 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:11.194964 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:11.194983 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:11.195320 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:11.195657 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:11.195659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:11.195661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:11.195808 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:11.196320 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:11.196325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:11.196552 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.196702 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.197078 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260924 07:16:11.199246 16766 heartbeater.cc:347] Connected to a master server at 127.15.127.190:39917
I20260924 07:16:11.199502 16766 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:11.200006 16766 heartbeater.cc:511] Master 127.15.127.190:39917 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:11.201105 16440 ts_manager.cc:194] Registered new tserver with Master: 98c41f56f12c43f8939289802cd55055 (127.15.127.130:34617)
I20260924 07:16:11.201663 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:37461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:11.220172 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39917
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:45351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:11.330516 16771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:11.330798 16771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:11.330873 16771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:11.334831 16771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:11.334920 16771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:11.335011 16771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:11.340319 16771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39917
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:11.341459 16771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:11.342590 16771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:11.352216 16776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:11.352288 16779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:11.352424 16771 server_base.cc:1034] running on GCE node
W20260924 07:16:11.352257 16777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:11.352892 16771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:11.353574 16771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:11.354774 16771 hybrid_clock.cc:648] HybridClock initialized: now 1790234171354754 us; error 32 us; skew 500 ppm
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:11.357486 16771 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:11.358279 16771 webserver.cc:460] Webserver started at http://127.15.127.131:43873/ using document root <none> and password file <none>
I20260924 07:16:11.358788 16771 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:11.358843 16771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:11.358997 16771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:11.360934 16771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "b89556fb5e1c476caa2e3ce9d9898cef"
format_stamp: "Formatted at 2026-09-24 07:16:11 on dist-test-slave-zx46"
server_key: "88b24dc6e021443d42a48f1254165d36"
server_key_iv: "03a961c45703acd718271461fc311aea"
server_key_version: "encryptionkey@0"
I20260924 07:16:11.361510 16771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "b89556fb5e1c476caa2e3ce9d9898cef"
format_stamp: "Formatted at 2026-09-24 07:16:11 on dist-test-slave-zx46"
server_key: "88b24dc6e021443d42a48f1254165d36"
server_key_iv: "03a961c45703acd718271461fc311aea"
server_key_version: "encryptionkey@0"
I20260924 07:16:11.364984 16771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:11.367384 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:11.368656 16771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:11.368769 16771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "b89556fb5e1c476caa2e3ce9d9898cef"
format_stamp: "Formatted at 2026-09-24 07:16:11 on dist-test-slave-zx46"
server_key: "88b24dc6e021443d42a48f1254165d36"
server_key_iv: "03a961c45703acd718271461fc311aea"
server_key_version: "encryptionkey@0"
I20260924 07:16:11.368856 16771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:11.387526 16771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:11.390339 16771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:11.390497 16771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:11.391021 16771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:11.391776 16771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:11.391820 16771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:11.391860 16771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:11.391881 16771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:11.400694 16771 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:45661
I20260924 07:16:11.400727 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:45661 every 8 connection(s)
I20260924 07:16:11.401629 16771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:11.406328 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16771
I20260924 07:16:11.406432 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:11.406704 15870 external_mini_cluster.cc:1428] Setting key a29867ecca0b6e17688ea5387e3c771c
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:11.414417 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.403382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:50119 (local address 127.15.127.190:39917)
0924 07:16:11.403520 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.403523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.404372 (+ 849us) server_negotiation.cc:407] Connection header received
0924 07:16:11.405339 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.405342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.405400 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.405507 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:11.406757 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.407340 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.408488 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.408721 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.408840 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.411196 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:11.411199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:11.411201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:11.411240 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:11.412632 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.413117 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.413119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.413121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.413192 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.413501 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.413503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.413504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.413673 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:11.413745 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.414185 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.414298 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:16:11.415196 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.403768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:39917 (local address 127.15.127.131:50119)
0924 07:16:11.404202 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.404260 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.405128 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.405678 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.405685 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.406040 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:11.406603 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.406621 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.407465 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.407468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.408362 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.408371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.408876 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.408880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.409070 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.409598 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:11.409613 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:11.411050 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:11.412724 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:11.412728 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:11.412745 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:11.412990 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:11.413286 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:11.413288 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:11.413289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:11.413418 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:11.413760 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:11.413765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:11.413979 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.414130 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.414367 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":116,"threads_started":1}
I20260924 07:16:11.416296 16900 heartbeater.cc:347] Connected to a master server at 127.15.127.190:39917
I20260924 07:16:11.416541 16900 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:11.417086 16900 heartbeater.cc:511] Master 127.15.127.190:39917 requested a full tablet report, sending...
I20260924 07:16:11.418076 16440 ts_manager.cc:194] Registered new tserver with Master: b89556fb5e1c476caa2e3ce9d9898cef (127.15.127.131:45661)
I20260924 07:16:11.418596 16440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:50119
I20260924 07:16:11.420313 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:16:11.431051 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.423865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.15.127.190:39917)
0924 07:16:11.424014 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.424018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.424239 (+ 221us) server_negotiation.cc:407] Connection header received
0924 07:16:11.424377 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.424379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.424418 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.424486 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:11.425187 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.425695 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.426465 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.426667 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.426743 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.427935 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:11.427938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:11.427940 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:11.427961 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:11.429525 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.429886 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.429888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.429890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.429960 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.430176 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.430178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.430179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.430389 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:11.430530 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.430739 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.430849 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:16:11.435570 16440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:11.438868 16440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:11.453130 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.447313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:45661 (local address 127.0.0.1:50640)
0924 07:16:11.448120 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.448170 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.448327 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.449151 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.449154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.449172 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:11.449359 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.449366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.450474 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.450478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.451544 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.451556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.452530 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.452533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.452638 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.452679 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.452817 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.452935 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":44,"spinlock_wait_cycles":58624,"thread_start_us":113,"threads_started":1}
I20260924 07:16:11.453162 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.447457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.15.127.131:45661)
0924 07:16:11.448606 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.448620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.448645 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:16:11.448723 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.448728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.448981 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.449164 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:11.449468 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.450357 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.451677 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.452408 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.452528 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.452658 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.452870 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.452988 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":126,"threads_started":1}
I20260924 07:16:11.453477 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.447476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43733 (local address 127.0.0.1:50332)
0924 07:16:11.448119 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.448169 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.448326 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.449133 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.449139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.449161 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:11.449356 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.449365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.450791 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.450795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.451959 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.451968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.453089 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.453093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.453191 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.453216 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.453283 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.453350 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260924 07:16:11.454051 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.448125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.15.127.129:43733)
0924 07:16:11.448648 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.448663 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.448685 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:16:11.448772 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.448779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.448972 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.449146 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:11.449491 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.450643 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.452080 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.452915 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.453609 (+ 694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.453749 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.453801 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.453891 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":122,"threads_started":1}
I20260924 07:16:11.454114 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.447692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:34617 (local address 127.0.0.1:40508)
0924 07:16:11.448505 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.448517 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.448590 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.449891 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.449894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.449912 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:11.449984 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.449988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.451294 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.451297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.452308 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.452315 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.453600 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.453604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.453828 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.453854 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.453933 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.454001 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"thread_start_us":99,"threads_started":1}
I20260924 07:16:11.454226 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.448111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.15.127.130:34617)
0924 07:16:11.449454 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.449468 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.449490 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:16:11.449560 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.449564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.449749 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.449892 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:11.450078 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.451150 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.452415 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.453307 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.453421 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.453535 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.453987 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.454085 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":93,"threads_started":1}
I20260924 07:16:11.456472 16700 tablet_service.cc:1467] Processing CreateTablet for tablet d03a0f67432e4f3498016126d20d5150 (DEFAULT_TABLE table=test-table [id=1e4a2a3714604c119e930864d160a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:11.456475 16834 tablet_service.cc:1467] Processing CreateTablet for tablet d03a0f67432e4f3498016126d20d5150 (DEFAULT_TABLE table=test-table [id=1e4a2a3714604c119e930864d160a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:11.457476 16834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03a0f67432e4f3498016126d20d5150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:11.457484 16700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03a0f67432e4f3498016126d20d5150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:11.458894 16565 tablet_service.cc:1467] Processing CreateTablet for tablet d03a0f67432e4f3498016126d20d5150 (DEFAULT_TABLE table=test-table [id=1e4a2a3714604c119e930864d160a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:11.459889 16565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d03a0f67432e4f3498016126d20d5150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:11.463770 16918 tablet_bootstrap.cc:492] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: Bootstrap starting.
I20260924 07:16:11.465270 16919 tablet_bootstrap.cc:492] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: Bootstrap starting.
I20260924 07:16:11.465945 16918 tablet_bootstrap.cc:654] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:11.466627 16920 tablet_bootstrap.cc:492] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f: Bootstrap starting.
I20260924 07:16:11.467182 16918 log.cc:824] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:11.467758 16919 tablet_bootstrap.cc:654] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:11.468707 16920 tablet_bootstrap.cc:654] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:11.469252 16919 log.cc:824] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:11.469873 16920 log.cc:824] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:11.470115 16918 tablet_bootstrap.cc:492] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: No bootstrap required, opened a new log
I20260924 07:16:11.470373 16918 ts_tablet_manager.cc:1397] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:16:11.472078 16919 tablet_bootstrap.cc:492] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: No bootstrap required, opened a new log
I20260924 07:16:11.472239 16920 tablet_bootstrap.cc:492] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f: No bootstrap required, opened a new log
I20260924 07:16:11.472388 16919 ts_tablet_manager.cc:1397] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:16:11.472458 16920 ts_tablet_manager.cc:1397] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 07:16:11.474381 16918 raft_consensus.cc:348] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.474670 16918 raft_consensus.cc:374] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:11.474726 16918 raft_consensus.cc:729] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89556fb5e1c476caa2e3ce9d9898cef, State: Initialized, Role: FOLLOWER
I20260924 07:16:11.475194 16918 consensus_queue.cc:260] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.476035 16920 raft_consensus.cc:348] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.476300 16920 raft_consensus.cc:374] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:11.476353 16920 raft_consensus.cc:729] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb6620dfd504313a5548831e123379f, State: Initialized, Role: FOLLOWER
I20260924 07:16:11.476275 16919 raft_consensus.cc:348] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.476492 16918 ts_tablet_manager.cc:1428] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:16:11.476536 16919 raft_consensus.cc:374] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:11.476591 16919 raft_consensus.cc:729] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98c41f56f12c43f8939289802cd55055, State: Initialized, Role: FOLLOWER
I20260924 07:16:11.476881 16900 heartbeater.cc:503] Master 127.15.127.190:39917 was elected leader, sending a full tablet report...
I20260924 07:16:11.476818 16920 consensus_queue.cc:260] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.477087 16919 consensus_queue.cc:260] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.477483 16631 heartbeater.cc:503] Master 127.15.127.190:39917 was elected leader, sending a full tablet report...
I20260924 07:16:11.477756 16920 ts_tablet_manager.cc:1428] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:16:11.477977 16919 ts_tablet_manager.cc:1428] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:16:11.478292 16766 heartbeater.cc:503] Master 127.15.127.190:39917 was elected leader, sending a full tablet report...
I20260924 07:16:11.613354 16925 raft_consensus.cc:482] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:11.613569 16925 raft_consensus.cc:504] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.614704 16925 leader_election.cc:288] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b89556fb5e1c476caa2e3ce9d9898cef (127.15.127.131:45661), 98c41f56f12c43f8939289802cd55055 (127.15.127.130:34617)
I20260924 07:16:11.620258 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.615098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:33327 (local address 127.15.127.130:34617)
0924 07:16:11.615242 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.615248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.615652 (+ 404us) server_negotiation.cc:407] Connection header received
0924 07:16:11.615730 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.615733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.615793 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.615888 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:11.616340 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.617149 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.618948 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.619577 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.619708 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.619764 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.620039 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.620107 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:11.620249 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.615098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60955 (local address 127.15.127.131:45661)
0924 07:16:11.615227 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.615232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.615573 (+ 341us) server_negotiation.cc:407] Connection header received
0924 07:16:11.615741 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.615745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.615789 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.615888 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:11.616340 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.617149 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.618948 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.619577 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.619708 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.619767 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.620039 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.620107 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:16:11.620337 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.615004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:45661 (local address 127.15.127.129:60955)
0924 07:16:11.615447 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.615467 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.615612 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.615920 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.615923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.615964 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:11.616201 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.616208 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.617268 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.617271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.618814 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.618826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.619723 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.619734 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.619871 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.619930 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.620009 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.620130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":259,"spinlock_wait_cycles":28928,"thread_start_us":135,"threads_started":1}
I20260924 07:16:11.620536 16927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.615004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:34617 (local address 127.15.127.129:33327)
0924 07:16:11.615545 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:11.615557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:11.615634 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:11.615920 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:11.615923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:11.615965 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:11.616206 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.616212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.617268 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.617271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.618814 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:11.618826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.619723 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.619727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:11.619874 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.619933 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 07:16:11.620009 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:16:11.620130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":285,"spinlock_wait_cycles":6400,"thread_start_us":104,"threads_started":1}
I20260924 07:16:11.621134 16854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03a0f67432e4f3498016126d20d5150" candidate_uuid: "4fb6620dfd504313a5548831e123379f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" is_pre_election: true
I20260924 07:16:11.621269 16720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03a0f67432e4f3498016126d20d5150" candidate_uuid: "4fb6620dfd504313a5548831e123379f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c41f56f12c43f8939289802cd55055" is_pre_election: true
I20260924 07:16:11.621496 16854 raft_consensus.cc:2454] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb6620dfd504313a5548831e123379f in term 0.
I20260924 07:16:11.621563 16720 raft_consensus.cc:2454] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb6620dfd504313a5548831e123379f in term 0.
I20260924 07:16:11.622006 16521 leader_election.cc:302] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb6620dfd504313a5548831e123379f, b89556fb5e1c476caa2e3ce9d9898cef; no voters:
I20260924 07:16:11.622321 16925 raft_consensus.cc:2781] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:11.622407 16925 raft_consensus.cc:482] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:11.622452 16925 raft_consensus.cc:3037] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:11.623677 16925 raft_consensus.cc:504] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.624053 16925 leader_election.cc:288] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [CANDIDATE]: Term 1 election: Requested vote from peers b89556fb5e1c476caa2e3ce9d9898cef (127.15.127.131:45661), 98c41f56f12c43f8939289802cd55055 (127.15.127.130:34617)
I20260924 07:16:11.624413 16854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03a0f67432e4f3498016126d20d5150" candidate_uuid: "4fb6620dfd504313a5548831e123379f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89556fb5e1c476caa2e3ce9d9898cef"
I20260924 07:16:11.624518 16720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d03a0f67432e4f3498016126d20d5150" candidate_uuid: "4fb6620dfd504313a5548831e123379f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98c41f56f12c43f8939289802cd55055"
I20260924 07:16:11.624547 16854 raft_consensus.cc:3037] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:11.624683 16720 raft_consensus.cc:3037] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:11.625734 16854 raft_consensus.cc:2454] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb6620dfd504313a5548831e123379f in term 1.
I20260924 07:16:11.626066 16521 leader_election.cc:302] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb6620dfd504313a5548831e123379f, b89556fb5e1c476caa2e3ce9d9898cef; no voters:
I20260924 07:16:11.626240 16925 raft_consensus.cc:2781] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:11.626238 16720 raft_consensus.cc:2454] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb6620dfd504313a5548831e123379f in term 1.
I20260924 07:16:11.626464 16925 raft_consensus.cc:686] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [term 1 LEADER]: Becoming Leader. State: Replica: 4fb6620dfd504313a5548831e123379f, State: Running, Role: LEADER
I20260924 07:16:11.626837 16925 consensus_queue.cc:237] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } }
I20260924 07:16:11.630100 16439 catalog_manager.cc:5495] T d03a0f67432e4f3498016126d20d5150 P 4fb6620dfd504313a5548831e123379f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fb6620dfd504313a5548831e123379f (127.15.127.129). New cstate: current_term: 1 leader_uuid: "4fb6620dfd504313a5548831e123379f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b89556fb5e1c476caa2e3ce9d9898cef" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fb6620dfd504313a5548831e123379f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98c41f56f12c43f8939289802cd55055" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34617 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:11.644964 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.635705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.15.127.129:43733)
0924 07:16:11.635823 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.635827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.636055 (+ 228us) server_negotiation.cc:407] Connection header received
0924 07:16:11.636240 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.636243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.636284 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.636372 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:11.637182 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.637784 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.638635 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.638861 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.638969 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.641428 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:11.641448 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:11.641462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:11.641484 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:11.643245 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.643677 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.643682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.643684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.643767 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.644047 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.644052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.644054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.644316 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:11.644539 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.644686 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.644807 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 07:16:11.652509 16901 tablet.cc:2369] T d03a0f67432e4f3498016126d20d5150 P b89556fb5e1c476caa2e3ce9d9898cef: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:11.670904 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.662558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.15.127.129:43733)
0924 07:16:11.662718 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.662723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.662861 (+ 138us) server_negotiation.cc:407] Connection header received
0924 07:16:11.663045 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.663048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.663089 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.663174 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:11.663851 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.664393 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.665310 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.665544 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.665652 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.667962 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:11.667966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:11.667968 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:11.667991 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:11.669247 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.669641 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.669646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.669648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.669745 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.670010 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.670015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.670016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.670178 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:11.670326 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.670603 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.670738 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260924 07:16:11.684453 16767 tablet.cc:2369] T d03a0f67432e4f3498016126d20d5150 P 98c41f56f12c43f8939289802cd55055: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:11.696302 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:11.688053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.15.127.129:43733)
0924 07:16:11.688245 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:11.688250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:11.688363 (+ 113us) server_negotiation.cc:407] Connection header received
0924 07:16:11.688532 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:11.688536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:11.688587 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:11.688674 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:11.689460 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.689962 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.690807 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:11.691032 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:11.691144 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:11.693557 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:11.693566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:11.693568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:11.693594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:11.694862 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.695269 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.695271 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.695272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.695340 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:11.695560 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:11.695562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:11.695563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:11.695712 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:11.695834 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:11.696017 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 07:16:11.696110 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 07:16:11.697773 16545 tablet_service.cc:2908] Rejecting scan request for tablet d03a0f67432e4f3498016126d20d5150: Uninitialized: clean time has not yet been initialized
I20260924 07:16:11.700525 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16502
I20260924 07:16:11.706323 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16637
I20260924 07:16:11.711658 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16771
I20260924 07:16:11.716758 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16408
2026-09-24T07:16:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5892 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16943](info): set up 2 sockets
Sep 24 07:16:11 dist-test-slave-zx46 krb5kdc[16943](info): commencing operation
krb5kdc: starting...
W20260924 07:16:12.760762 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:12 dist-test-slave-zx46 krb5kdc[16943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:12.886229 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:34729
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:34729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:12.995182 16956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:12.995512 16956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:12.995592 16956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:12.999698 16956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:12.999799 16956 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:16:12.999832 16956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:12.999876 16956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:12.999913 16956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:12.999949 16956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:13.005345 16956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:34729
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:34729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:13.006422 16956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:13.007447 16956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:13.016036 16961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:13.016044 16964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:13.016184 16956 server_base.cc:1034] running on GCE node
W20260924 07:16:13.016122 16962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:13.016916 16956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:13.018085 16956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:13.019251 16956 hybrid_clock.cc:648] HybridClock initialized: now 1790234173019233 us; error 31 us; skew 500 ppm
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:13.022018 16956 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:13.022919 16956 webserver.cc:460] Webserver started at http://127.15.127.190:39511/ using document root <none> and password file <none>
I20260924 07:16:13.023445 16956 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:13.023504 16956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:13.023666 16956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:13.025741 16956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "bf5f47ecd05a4285845a6544792eb6d0"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "729015270953eef33e997199ce8bf490"
server_key_iv: "f83b967094a6c47c15fee8daa19333ef"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.026265 16956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "bf5f47ecd05a4285845a6544792eb6d0"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "729015270953eef33e997199ce8bf490"
server_key_iv: "f83b967094a6c47c15fee8daa19333ef"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.029853 16956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:13.032230 16971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.033537 16956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:13.033638 16956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "bf5f47ecd05a4285845a6544792eb6d0"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "729015270953eef33e997199ce8bf490"
server_key_iv: "f83b967094a6c47c15fee8daa19333ef"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.033727 16956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:13.044737 16956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:13.047610 16956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:13.047761 16956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:13.055431 16956 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:34729
I20260924 07:16:13.055452 17033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:34729 every 8 connection(s)
I20260924 07:16:13.056370 16956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:13.059111 17034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:13.062100 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 16956
I20260924 07:16:13.062216 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:13.062464 15870 external_mini_cluster.cc:1428] Setting key 58ba3f0d2379c4d914b35bb3e4a1deba
I20260924 07:16:13.064956 17034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: Bootstrap starting.
I20260924 07:16:13.067094 17034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:13.067790 17034 log.cc:824] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:13.070047 17034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: No bootstrap required, opened a new log
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234172, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:13.074433 17034 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5f47ecd05a4285845a6544792eb6d0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34729 } }
I20260924 07:16:13.074724 17034 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:13.074805 17034 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf5f47ecd05a4285845a6544792eb6d0, State: Initialized, Role: FOLLOWER
I20260924 07:16:13.075290 17034 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5f47ecd05a4285845a6544792eb6d0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34729 } }
I20260924 07:16:13.075451 17034 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:13.075567 17034 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:13.075672 17034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:13.076654 17034 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5f47ecd05a4285845a6544792eb6d0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34729 } }
I20260924 07:16:13.077045 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.063881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.15.127.190:34729)
0924 07:16:13.064299 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.064314 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.064348 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:16:13.064944 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.064961 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.065376 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.065701 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.066497 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.067529 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.068578 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.069243 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.069419 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.071987 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.071993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.072012 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.072042 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.073987 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.074404 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.074410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.074416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.074589 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.074922 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.074926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.074927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.075392 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.075601 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.075950 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.076233 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260924 07:16:13.076994 17034 leader_election.cc:302] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf5f47ecd05a4285845a6544792eb6d0; no voters:
I20260924 07:16:13.077517 17034 leader_election.cc:288] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:13.077584 17039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:13.077828 17039 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [term 1 LEADER]: Becoming Leader. State: Replica: bf5f47ecd05a4285845a6544792eb6d0, State: Running, Role: LEADER
I20260924 07:16:13.078848 17034 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:13.079329 17039 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5f47ecd05a4285845a6544792eb6d0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34729 } }
I20260924 07:16:13.080590 17039 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf5f47ecd05a4285845a6544792eb6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5f47ecd05a4285845a6544792eb6d0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34729 } } }
I20260924 07:16:13.080713 17039 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:13.081143 17040 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf5f47ecd05a4285845a6544792eb6d0. Latest consensus state: current_term: 1 leader_uuid: "bf5f47ecd05a4285845a6544792eb6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf5f47ecd05a4285845a6544792eb6d0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34729 } } }
I20260924 07:16:13.081326 17040 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:13.081544 17053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:13.084631 17053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:13.089928 17053 catalog_manager.cc:1347] Generated new cluster ID: 44245a8ccabe4a6fae051a31f2a69a1d
I20260924 07:16:13.090094 17053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:13.106618 17053 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:13.107918 17053 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:13.113747 17053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: Generated new TSK 0
I20260924 07:16:13.114553 17053 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:13.147523 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34729
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:13.257725 17060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:13.257978 17060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:13.258023 17060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:13.261912 17060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:13.261969 17060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:13.262007 17060 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:13.262038 17060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:13.267246 17060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:13.268281 17060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:13.269284 17060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:13.278723 17066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:13.278712 17065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:13.278764 17060 server_base.cc:1034] running on GCE node
W20260924 07:16:13.278869 17068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:13.279397 17060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:13.280108 17060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:13.281299 17060 hybrid_clock.cc:648] HybridClock initialized: now 1790234173281283 us; error 29 us; skew 500 ppm
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:13.283991 17060 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:13.285081 17060 webserver.cc:460] Webserver started at http://127.15.127.129:36471/ using document root <none> and password file <none>
I20260924 07:16:13.285722 17060 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:13.285830 17060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:13.286101 17060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:13.288365 17060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "e39720af34ec43c5afe592d9dce9b6c0"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "897d81f4cd9c0d322d101d1214220b1c"
server_key_iv: "0d2825d07bc808786c94dc05cfa5ec64"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.289060 17060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "e39720af34ec43c5afe592d9dce9b6c0"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "897d81f4cd9c0d322d101d1214220b1c"
server_key_iv: "0d2825d07bc808786c94dc05cfa5ec64"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.292990 17060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:16:13.295604 17075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.296754 17060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:13.296911 17060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "e39720af34ec43c5afe592d9dce9b6c0"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "897d81f4cd9c0d322d101d1214220b1c"
server_key_iv: "0d2825d07bc808786c94dc05cfa5ec64"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.297040 17060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:13.306237 17060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:13.308904 17060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:13.309137 17060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:13.310475 17060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:13.310550 17060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.310623 17060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:13.310667 17060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:13.321429 17060 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:45135
I20260924 07:16:13.321645 17190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:45135 every 8 connection(s)
I20260924 07:16:13.322397 17060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:13.323660 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17060
I20260924 07:16:13.323786 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:13.324090 15870 external_mini_cluster.cc:1428] Setting key a357abdee7b62718073a37383e082136
I20260924 07:16:13.325500 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.311350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:46213 (local address 127.15.127.190:34729)
0924 07:16:13.311473 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.311477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.312478 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 07:16:13.313610 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.313614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.313679 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.313746 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.315543 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.316448 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.318290 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.318644 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.318774 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.321580 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.321584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.321587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.321618 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.323271 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.323873 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.323879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.323884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.323965 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.324351 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.324353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.324354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.324562 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.324675 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.325209 (+ 534us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.325334 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:16:13.326242 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.311878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.15.127.129:46213)
0924 07:16:13.312335 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.312376 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.313329 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.314220 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.314228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.314695 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:13.315360 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.315382 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.316587 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.316591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.318122 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.318133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.318790 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.318793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.318926 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.319556 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:13.319577 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:13.321377 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:13.323386 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.323391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.323405 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.323751 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.324061 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.324064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.324066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.324251 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.324672 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:13.324677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:13.324905 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.325136 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.325450 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260924 07:16:13.327454 17191 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34729
I20260924 07:16:13.327598 17191 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:13.328058 17191 heartbeater.cc:511] Master 127.15.127.190:34729 requested a full tablet report, sending...
I20260924 07:16:13.329732 16987 ts_manager.cc:194] Registered new tserver with Master: e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135)
I20260924 07:16:13.331097 16987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:46213
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:13.337672 17198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.329723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.15.127.129:43761)
0924 07:16:13.330057 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.330074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.330170 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.330425 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.330429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.330682 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:13.330844 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.330855 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.331820 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.331826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.332924 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.332939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.333495 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.333499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.333601 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.334081 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:13.334097 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:13.334535 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:13.336282 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.336284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.336286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.336462 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.336768 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.336770 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.336772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.336845 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.337251 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:13.337253 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:13.337325 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.337407 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.337503 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":93,"threads_started":1}
I20260924 07:16:13.337801 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.329776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:43761 (local address 127.15.127.190:34729)
0924 07:16:13.329882 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.329886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.330153 (+ 267us) server_negotiation.cc:407] Connection header received
0924 07:16:13.330260 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.330263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.330303 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.330382 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.330973 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.331649 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.333103 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.333365 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.333448 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.334676 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.334679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.334681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.334702 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.336167 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.336544 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.336547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.336548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.336615 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.336937 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.336939 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.336941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.337154 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.337250 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.337504 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.337592 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":44}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:13.354228 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34729
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:13.468258 17200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:13.468531 17200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:13.468583 17200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:13.472622 17200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:13.472687 17200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:13.472725 17200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:13.472776 17200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:13.479355 17200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:13.480500 17200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:13.481626 17200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:13.491339 17208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:13.491400 17206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:13.491466 17205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:13.491528 17200 server_base.cc:1034] running on GCE node
I20260924 07:16:13.492043 17200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:13.492679 17200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:13.493886 17200 hybrid_clock.cc:648] HybridClock initialized: now 1790234173493875 us; error 34 us; skew 500 ppm
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:13.496659 17200 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:13.497624 17200 webserver.cc:460] Webserver started at http://127.15.127.130:39973/ using document root <none> and password file <none>
I20260924 07:16:13.498188 17200 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:13.498277 17200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:13.498504 17200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:13.500625 17200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "81654a93e916468a978ee4d1a221ab2b"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "38ccca707c562dbff9de0cdbe2cf73be"
server_key_iv: "409b0981488d8779e8b943abdf8fa09c"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.501281 17200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "81654a93e916468a978ee4d1a221ab2b"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "38ccca707c562dbff9de0cdbe2cf73be"
server_key_iv: "409b0981488d8779e8b943abdf8fa09c"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.504933 17200 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:16:13.507318 17215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.508514 17200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:13.508649 17200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "81654a93e916468a978ee4d1a221ab2b"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "38ccca707c562dbff9de0cdbe2cf73be"
server_key_iv: "409b0981488d8779e8b943abdf8fa09c"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.508769 17200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:13.519414 17200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:13.522635 17200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:13.522833 17200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:13.524400 17200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:13.524474 17200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.524541 17200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:13.524595 17200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.534322 17200 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:38565
I20260924 07:16:13.534399 17330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:38565 every 8 connection(s)
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:13.535318 17200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:13.538947 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.525144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:51121 (local address 127.15.127.190:34729)
0924 07:16:13.525298 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.525304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.526211 (+ 907us) server_negotiation.cc:407] Connection header received
0924 07:16:13.527346 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.527351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.527417 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.527530 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.529046 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.529953 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.531838 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.532065 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.532170 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.535337 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.535341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.535345 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.535379 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.537140 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.537655 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.537658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.537659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.537734 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.537993 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.537995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.537996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.538163 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.538323 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.538737 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.538830 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:16:13.539759 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.525595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.15.127.130:51121)
0924 07:16:13.526043 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.526093 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.527088 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.527718 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.527728 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.528146 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:13.528851 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.528874 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.530146 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.530153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.531651 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.531703 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.532223 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.532228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.532460 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.533237 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:13.533263 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:13.535155 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:13.537253 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.537258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.537268 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.537570 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.537796 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.537798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.537799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.537912 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.538312 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:13.538317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:13.538534 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.538686 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.538924 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":138,"threads_started":1}
I20260924 07:16:13.540431 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17200
I20260924 07:16:13.540553 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:13.540808 15870 external_mini_cluster.cc:1428] Setting key 12e6e05a567c0795d3f426f1c8e55994
I20260924 07:16:13.541208 17331 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34729
I20260924 07:16:13.541376 17331 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:13.541926 17331 heartbeater.cc:511] Master 127.15.127.190:34729 requested a full tablet report, sending...
I20260924 07:16:13.543133 16988 ts_manager.cc:194] Registered new tserver with Master: 81654a93e916468a978ee4d1a221ab2b (127.15.127.130:38565)
I20260924 07:16:13.543825 16988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:51121
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:13.551043 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.543399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.15.127.130:40967)
0924 07:16:13.543733 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.543750 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.543849 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.544120 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.544123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.544317 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:13.544413 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.544418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.545523 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.545526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.546195 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.546204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.546648 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.546652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.546756 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.547309 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:13.547330 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:13.547641 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:13.549427 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.549433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.549435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.549688 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.550010 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.550012 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.550014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.550089 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.550600 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:13.550602 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:13.550675 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.550767 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.550873 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":101,"threads_started":1}
I20260924 07:16:13.551158 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.543370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:40967 (local address 127.15.127.190:34729)
0924 07:16:13.543474 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.543478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.543833 (+ 355us) server_negotiation.cc:407] Connection header received
0924 07:16:13.543973 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.543976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.544018 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.544105 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.544538 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.545389 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.546318 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.546539 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.546646 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.547783 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.547786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.547789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.547812 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.549322 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.549810 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.549817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.549821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.549913 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.550185 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.550189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.550192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.550500 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.550615 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.550846 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.550970 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:13.569837 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34729
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:35181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:13.679611 17340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:13.679930 17340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:13.680006 17340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:13.684443 17340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:13.684518 17340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:13.684561 17340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:13.684595 17340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:13.690444 17340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:35181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:34729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:13.691524 17340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:13.692553 17340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:13.702143 17348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:13.702232 17345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:13.702150 17346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:13.702323 17340 server_base.cc:1034] running on GCE node
I20260924 07:16:13.702867 17340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:13.703514 17340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:13.704696 17340 hybrid_clock.cc:648] HybridClock initialized: now 1790234173704669 us; error 49 us; skew 500 ppm
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:13.707496 17340 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:13.708366 17340 webserver.cc:460] Webserver started at http://127.15.127.131:41279/ using document root <none> and password file <none>
I20260924 07:16:13.708916 17340 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:13.709007 17340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:13.709288 17340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:13.711289 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "7051405a739d4851a5900f1f89c6d3d5"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "9cd40b564aab532ffb089aa176a454b5"
server_key_iv: "97103174bfb6072caea5b048bf34fd15"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.711897 17340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "7051405a739d4851a5900f1f89c6d3d5"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "9cd40b564aab532ffb089aa176a454b5"
server_key_iv: "97103174bfb6072caea5b048bf34fd15"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.715646 17340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:16:13.717975 17355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.719103 17340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:13.719242 17340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "7051405a739d4851a5900f1f89c6d3d5"
format_stamp: "Formatted at 2026-09-24 07:16:13 on dist-test-slave-zx46"
server_key: "9cd40b564aab532ffb089aa176a454b5"
server_key_iv: "97103174bfb6072caea5b048bf34fd15"
server_key_version: "encryptionkey@0"
I20260924 07:16:13.719348 17340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:13.732966 17340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:13.736439 17340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:13.736644 17340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:13.738297 17340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:13.738368 17340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:13.738440 17340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:13.738471 17340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:13.749509 17340 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:38765
I20260924 07:16:13.750392 17340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:13.751195 17470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:38765 every 8 connection(s)
I20260924 07:16:13.752084 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.739060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:39403 (local address 127.15.127.190:34729)
0924 07:16:13.739195 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.739200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.740242 (+ 1042us) server_negotiation.cc:407] Connection header received
0924 07:16:13.741379 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.741384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.741447 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.741511 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.743249 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.743862 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.745033 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.745280 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.745400 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.748259 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.748263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.748265 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.748301 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.750027 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.750534 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.750539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.750541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.750631 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.751038 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.751042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.751044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.751296 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.751391 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.751863 (+ 472us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.751963 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:16:13.752965 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.739616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.15.127.131:39403)
0924 07:16:13.740093 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.740136 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.741131 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.741999 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.742006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.742476 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:13.743100 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.743116 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.743991 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.743994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.744881 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.744889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.745447 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.745452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.745654 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.746357 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:13.746376 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:13.748082 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:13.750146 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.750152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.750168 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.750427 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.750728 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.750731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.750733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.750926 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.751414 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:13.751425 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:13.751644 (+ 219us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.751811 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.752058 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":127,"threads_started":1}
I20260924 07:16:13.754293 17471 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34729
I20260924 07:16:13.754447 17471 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:13.754925 17471 heartbeater.cc:511] Master 127.15.127.190:34729 requested a full tablet report, sending...
I20260924 07:16:13.755954 16987 ts_manager.cc:194] Registered new tserver with Master: 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765)
I20260924 07:16:13.756073 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17340
I20260924 07:16:13.756179 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:13.756484 15870 external_mini_cluster.cc:1428] Setting key b6fe217c60817905d122b08b5c8e7e9f
I20260924 07:16:13.756740 16987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:39403
I20260924 07:16:13.759333 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:16:13.763972 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.756513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:42023 (local address 127.15.127.190:34729)
0924 07:16:13.756612 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.756616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.756837 (+ 221us) server_negotiation.cc:407] Connection header received
0924 07:16:13.756963 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.756967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.757056 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.757123 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.757778 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.758498 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.759362 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.759564 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.759640 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.760903 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.760906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.760909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.760929 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.762405 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.762828 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.762832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.762834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.762916 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.763186 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.763191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.763192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.763364 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.763471 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.763730 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.763836 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:16:13.763986 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.756434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.15.127.131:42023)
0924 07:16:13.756756 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.756773 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.756872 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.757237 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.757241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.757479 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:13.757623 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.757630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.758631 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.758633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.759261 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.759273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.759667 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.759671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.759777 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.760286 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:13.760306 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:13.760774 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:13.762519 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.762524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.762526 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.762726 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.763006 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:13.763011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:13.763013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:13.763098 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:13.763460 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:13.763463 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:13.763556 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.763659 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.763803 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":2,"thread_start_us":102,"threads_started":1}
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:16:13 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:13.785147 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.777437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.15.127.190:34729)
0924 07:16:13.777561 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.777564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.777576 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:13.777615 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.777617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.777655 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.777725 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:13.778432 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.778967 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.779812 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.780064 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.780142 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.782519 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:13.782522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:13.782524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:13.782547 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:13.783774 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.784127 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.784129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.784131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.784199 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:13.784414 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:13.784416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:13.784417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:13.784573 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:13.784665 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.784866 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.784956 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:16:13.788074 16987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:13.791095 16987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:13.805572 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.799452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.15.127.131:38765)
0924 07:16:13.800253 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.800266 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.800290 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:16:13.800364 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.800370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.800586 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.801185 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:13.801339 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.802379 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.803989 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.804813 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.804969 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.805137 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.805315 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.805420 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":114,"threads_started":1}
I20260924 07:16:13.805562 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.799345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38765 (local address 127.0.0.1:48206)
0924 07:16:13.799791 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.799837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.799959 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.800794 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.800800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.800824 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:13.800979 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.800986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.802486 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.802492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.803865 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.803874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.804941 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.804945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.805095 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.805140 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.805272 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.805399 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":97,"thread_start_us":72,"threads_started":1}
I20260924 07:16:13.805961 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.800073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38565 (local address 127.0.0.1:40868)
0924 07:16:13.801184 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.801206 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.801340 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.801965 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.801968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.801985 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:13.802061 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.802066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.803425 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.803428 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.804417 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.804425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.805547 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.805550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.805647 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.805673 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.805760 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.805828 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":3,"thread_start_us":82,"threads_started":1}
I20260924 07:16:13.806360 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.800617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.15.127.130:38565)
0924 07:16:13.801023 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.801034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.801271 (+ 237us) server_negotiation.cc:407] Connection header received
0924 07:16:13.801479 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.801484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.801695 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.801813 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:13.802188 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.803302 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.804527 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.805257 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.805381 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.805489 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.806135 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.806230 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":103,"threads_started":1}
I20260924 07:16:13.806507 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.800313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58126 (local address 127.15.127.129:45135)
0924 07:16:13.800797 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:13.800812 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:13.800833 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:16:13.800900 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:13.800903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:13.801118 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:13.801285 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:13.801460 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.802381 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.804109 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.804834 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.805116 (+ 282us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.805232 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:13.805988 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 07:16:13.806280 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":85,"threads_started":1}
I20260924 07:16:13.805977 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:13.799401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45135 (local address 127.0.0.1:58126)
0924 07:16:13.800446 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:13.800460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:13.800543 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:13.801257 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:13.801259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:13.801277 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:13.801375 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.801381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.802518 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.802521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.803987 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:13.803997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:13.804968 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:13.804971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:13.805347 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:13.805369 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:16:13.805442 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:16:13.805507 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":393,"thread_start_us":120,"threads_started":1}
I20260924 07:16:13.808895 17265 tablet_service.cc:1467] Processing CreateTablet for tablet e5d16fe77113479ca71d104b25929dbb (DEFAULT_TABLE table=test-table [id=b70affb3116f42e088e0396149a59fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:13.808853 17405 tablet_service.cc:1467] Processing CreateTablet for tablet e5d16fe77113479ca71d104b25929dbb (DEFAULT_TABLE table=test-table [id=b70affb3116f42e088e0396149a59fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:13.808948 17125 tablet_service.cc:1467] Processing CreateTablet for tablet e5d16fe77113479ca71d104b25929dbb (DEFAULT_TABLE table=test-table [id=b70affb3116f42e088e0396149a59fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:13.810045 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d16fe77113479ca71d104b25929dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:13.810042 17265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d16fe77113479ca71d104b25929dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:13.810092 17125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5d16fe77113479ca71d104b25929dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:13.817335 17494 tablet_bootstrap.cc:492] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Bootstrap starting.
I20260924 07:16:13.818688 17496 tablet_bootstrap.cc:492] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Bootstrap starting.
I20260924 07:16:13.819974 17494 tablet_bootstrap.cc:654] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:13.820497 17496 tablet_bootstrap.cc:654] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:13.820784 17494 log.cc:824] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:13.821364 17495 tablet_bootstrap.cc:492] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Bootstrap starting.
I20260924 07:16:13.821376 17496 log.cc:824] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:13.822890 17494 tablet_bootstrap.cc:492] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: No bootstrap required, opened a new log
I20260924 07:16:13.823184 17494 ts_tablet_manager.cc:1397] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:16:13.823282 17496 tablet_bootstrap.cc:492] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: No bootstrap required, opened a new log
I20260924 07:16:13.823477 17496 ts_tablet_manager.cc:1397] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:16:13.823529 17495 tablet_bootstrap.cc:654] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:13.824380 17495 log.cc:824] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:13.826462 17495 tablet_bootstrap.cc:492] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: No bootstrap required, opened a new log
I20260924 07:16:13.826683 17495 ts_tablet_manager.cc:1397] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:16:13.826982 17496 raft_consensus.cc:348] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:13.827276 17496 raft_consensus.cc:374] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:13.827353 17496 raft_consensus.cc:729] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81654a93e916468a978ee4d1a221ab2b, State: Initialized, Role: FOLLOWER
I20260924 07:16:13.827275 17494 raft_consensus.cc:348] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:13.827580 17494 raft_consensus.cc:374] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:13.827718 17494 raft_consensus.cc:729] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39720af34ec43c5afe592d9dce9b6c0, State: Initialized, Role: FOLLOWER
I20260924 07:16:13.827857 17496 consensus_queue.cc:260] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:13.828133 17494 consensus_queue.cc:260] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:13.828562 17331 heartbeater.cc:503] Master 127.15.127.190:34729 was elected leader, sending a full tablet report...
I20260924 07:16:13.828900 17496 ts_tablet_manager.cc:1428] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:16:13.829955 17191 heartbeater.cc:503] Master 127.15.127.190:34729 was elected leader, sending a full tablet report...
I20260924 07:16:13.830252 17494 ts_tablet_manager.cc:1428] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:16:13.830572 17495 raft_consensus.cc:348] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:13.830752 17495 raft_consensus.cc:374] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:13.830785 17495 raft_consensus.cc:729] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7051405a739d4851a5900f1f89c6d3d5, State: Initialized, Role: FOLLOWER
I20260924 07:16:13.831243 17495 consensus_queue.cc:260] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:13.832067 17471 heartbeater.cc:503] Master 127.15.127.190:34729 was elected leader, sending a full tablet report...
I20260924 07:16:13.832211 17495 ts_tablet_manager.cc:1428] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 07:16:14.001485 17472 tablet.cc:2369] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:14.036581 17332 tablet.cc:2369] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:14.057680 17500 raft_consensus.cc:482] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:14.057897 17500 raft_consensus.cc:504] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:14.059062 17500 leader_election.cc:288] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765), e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135)
I20260924 07:16:14.063472 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.059713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:44423 (local address 127.15.127.129:45135)
0924 07:16:14.059829 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.059835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.059853 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:16:14.059908 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.059911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.059958 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.060044 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:14.060378 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.061194 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.062518 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.062916 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.063051 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.063106 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.063274 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.063343 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:16:14.063491 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.059330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45135 (local address 127.15.127.130:44423)
0924 07:16:14.059734 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.059746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.059816 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.060071 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.060074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.060094 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:14.060243 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.060251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.061304 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.061307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.062395 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.062404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.063041 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.063045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.063140 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.063169 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.063259 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.063340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260924 07:16:14.063719 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.059444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:55541 (local address 127.15.127.131:38765)
0924 07:16:14.059566 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.059570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.059687 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:16:14.059911 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.059921 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.059960 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.059999 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:14.060403 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.060908 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.062434 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.063055 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.063167 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.063213 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.063519 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.063603 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:16:14.064049 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.059278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38765 (local address 127.15.127.130:55541)
0924 07:16:14.059599 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.059616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.059774 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.060153 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.060156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.060167 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:14.060309 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.060318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.061049 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.061053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.062294 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.062305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.063272 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.063276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.063385 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.063410 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.063845 (+ 435us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.063921 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":44,"thread_start_us":86,"threads_started":1}
I20260924 07:16:14.064330 17145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d16fe77113479ca71d104b25929dbb" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39720af34ec43c5afe592d9dce9b6c0" is_pre_election: true
I20260924 07:16:14.064633 17145 raft_consensus.cc:2454] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 0.
I20260924 07:16:14.064627 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d16fe77113479ca71d104b25929dbb" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7051405a739d4851a5900f1f89c6d3d5" is_pre_election: true
I20260924 07:16:14.064915 17425 raft_consensus.cc:2454] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 0.
I20260924 07:16:14.065197 17216 leader_election.cc:302] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81654a93e916468a978ee4d1a221ab2b, e39720af34ec43c5afe592d9dce9b6c0; no voters:
I20260924 07:16:14.065477 17500 raft_consensus.cc:2781] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:14.065605 17500 raft_consensus.cc:482] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:14.065668 17500 raft_consensus.cc:3037] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.066684 17500 raft_consensus.cc:504] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:14.067075 17500 leader_election.cc:288] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 election: Requested vote from peers 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765), e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135)
I20260924 07:16:14.067355 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d16fe77113479ca71d104b25929dbb" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7051405a739d4851a5900f1f89c6d3d5"
I20260924 07:16:14.067435 17145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5d16fe77113479ca71d104b25929dbb" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39720af34ec43c5afe592d9dce9b6c0"
I20260924 07:16:14.067478 17425 raft_consensus.cc:3037] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.067559 17145 raft_consensus.cc:3037] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.068852 17145 raft_consensus.cc:2454] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 1.
I20260924 07:16:14.069175 17216 leader_election.cc:302] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81654a93e916468a978ee4d1a221ab2b, e39720af34ec43c5afe592d9dce9b6c0; no voters:
I20260924 07:16:14.069298 17425 raft_consensus.cc:2454] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 1.
I20260924 07:16:14.069370 17500 raft_consensus.cc:2781] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:14.069536 17500 raft_consensus.cc:686] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 1 LEADER]: Becoming Leader. State: Replica: 81654a93e916468a978ee4d1a221ab2b, State: Running, Role: LEADER
I20260924 07:16:14.069852 17500 consensus_queue.cc:237] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } }
I20260924 07:16:14.072988 16988 catalog_manager.cc:5495] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 81654a93e916468a978ee4d1a221ab2b (127.15.127.130). New cstate: current_term: 1 leader_uuid: "81654a93e916468a978ee4d1a221ab2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } health_report { overall_health: HEALTHY } } }
W20260924 07:16:14.073728 17192 tablet.cc:2369] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:14.106992 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.103204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.15.127.130:38565)
0924 07:16:14.103353 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.103358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.103372 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:14.103590 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.103592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.103649 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.103755 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:14.104016 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.104533 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.105503 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.105723 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.105830 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.106172 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.106575 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.106703 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.106786 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.106842 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:16:14.112934 17425 raft_consensus.cc:1261] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Refusing update from remote peer 81654a93e916468a978ee4d1a221ab2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:14.112934 17145 raft_consensus.cc:1261] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Refusing update from remote peer 81654a93e916468a978ee4d1a221ab2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:14.113649 17505 consensus_queue.cc:1035] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:14.113838 17500 consensus_queue.cc:1035] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:14.120925 17509 mvcc.cc:205] Tried to move back new op lower bound from 7332799177159110656 to 7332799176992600064. Current Snapshot: MvccSnapshot[applied={T|T < 7332799177159110656}]
I20260924 07:16:14.122710 17510 mvcc.cc:205] Tried to move back new op lower bound from 7332799177159110656 to 7332799176992600064. Current Snapshot: MvccSnapshot[applied={T|T < 7332799177159110656}]
I20260924 07:16:14.132537 16986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:14.132759 16986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:14.135282 16986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: Sending DeleteTablet for 3 replicas of tablet e5d16fe77113479ca71d104b25929dbb
I20260924 07:16:14.136135 17125 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d16fe77113479ca71d104b25929dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:14 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:58126
I20260924 07:16:14.136137 17265 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d16fe77113479ca71d104b25929dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:14 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40868
I20260924 07:16:14.136147 17405 tablet_service.cc:1514] Processing DeleteTablet for tablet e5d16fe77113479ca71d104b25929dbb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:14 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:48206
I20260924 07:16:14.136700 17518 tablet_replica.cc:330] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: stopping tablet replica
I20260924 07:16:14.136829 17519 tablet_replica.cc:330] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: stopping tablet replica
I20260924 07:16:14.137110 17518 raft_consensus.cc:2229] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:14.137243 17519 raft_consensus.cc:2229] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:14.137548 17518 raft_consensus.cc:2258] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:14.137672 17519 raft_consensus.cc:2258] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:14.137612 16986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:14.138055 16986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:14.138059 17520 tablet_replica.cc:330] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: stopping tablet replica
I20260924 07:16:14.138383 17520 raft_consensus.cc:2229] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:14.138653 17520 raft_consensus.cc:2258] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:14.139094 17518 ts_tablet_manager.cc:1905] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:14.139106 17519 ts_tablet_manager.cc:1905] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:14.140131 17520 ts_tablet_manager.cc:1905] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:14.141929 17518 ts_tablet_manager.cc:1918] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:14.142134 17518 log.cc:1196] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/e5d16fe77113479ca71d104b25929dbb
I20260924 07:16:14.142514 17518 ts_tablet_manager.cc:1939] T e5d16fe77113479ca71d104b25929dbb P e39720af34ec43c5afe592d9dce9b6c0: Deleting consensus metadata
I20260924 07:16:14.142831 17520 ts_tablet_manager.cc:1918] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:14.142930 17520 log.cc:1196] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/e5d16fe77113479ca71d104b25929dbb
I20260924 07:16:14.143040 17519 ts_tablet_manager.cc:1918] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:14.143139 17519 log.cc:1196] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/e5d16fe77113479ca71d104b25929dbb
I20260924 07:16:14.143260 17520 ts_tablet_manager.cc:1939] T e5d16fe77113479ca71d104b25929dbb P 7051405a739d4851a5900f1f89c6d3d5: Deleting consensus metadata
I20260924 07:16:14.143450 17519 ts_tablet_manager.cc:1939] T e5d16fe77113479ca71d104b25929dbb P 81654a93e916468a978ee4d1a221ab2b: Deleting consensus metadata
I20260924 07:16:14.144433 16975 catalog_manager.cc:4841] TS 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765): tablet e5d16fe77113479ca71d104b25929dbb (table test-table [id=b70affb3116f42e088e0396149a59fb2]) successfully deleted
I20260924 07:16:14.144622 16975 catalog_manager.cc:4841] TS 81654a93e916468a978ee4d1a221ab2b (127.15.127.130:38565): tablet e5d16fe77113479ca71d104b25929dbb (table test-table [id=b70affb3116f42e088e0396149a59fb2]) successfully deleted
I20260924 07:16:14.144758 17405 tablet_service.cc:1467] Processing CreateTablet for tablet f99d74337d664a74a08a676620b1a0f2 (DEFAULT_TABLE table=test-table [id=7a0ec225e1f94b819ee8d114f12d11df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:14.144932 16972 catalog_manager.cc:4841] TS e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135): tablet e5d16fe77113479ca71d104b25929dbb (table test-table [id=b70affb3116f42e088e0396149a59fb2]) successfully deleted
I20260924 07:16:14.145066 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99d74337d664a74a08a676620b1a0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:14.144981 17265 tablet_service.cc:1467] Processing CreateTablet for tablet f99d74337d664a74a08a676620b1a0f2 (DEFAULT_TABLE table=test-table [id=7a0ec225e1f94b819ee8d114f12d11df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:14.145299 17265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99d74337d664a74a08a676620b1a0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:14.145418 17125 tablet_service.cc:1467] Processing CreateTablet for tablet f99d74337d664a74a08a676620b1a0f2 (DEFAULT_TABLE table=test-table [id=7a0ec225e1f94b819ee8d114f12d11df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:14.145656 17125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99d74337d664a74a08a676620b1a0f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:14.147531 17496 tablet_bootstrap.cc:492] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Bootstrap starting.
I20260924 07:16:14.147593 17495 tablet_bootstrap.cc:492] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Bootstrap starting.
I20260924 07:16:14.147907 17494 tablet_bootstrap.cc:492] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Bootstrap starting.
I20260924 07:16:14.148638 17496 tablet_bootstrap.cc:654] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:14.148720 17495 tablet_bootstrap.cc:654] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:14.148962 17494 tablet_bootstrap.cc:654] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:14.150151 17496 tablet_bootstrap.cc:492] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: No bootstrap required, opened a new log
I20260924 07:16:14.150269 17496 ts_tablet_manager.cc:1397] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:14.150745 17496 raft_consensus.cc:348] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.150866 17496 raft_consensus.cc:374] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:14.150906 17496 raft_consensus.cc:729] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81654a93e916468a978ee4d1a221ab2b, State: Initialized, Role: FOLLOWER
I20260924 07:16:14.151033 17496 consensus_queue.cc:260] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.151232 17495 tablet_bootstrap.cc:492] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: No bootstrap required, opened a new log
I20260924 07:16:14.151296 17496 ts_tablet_manager.cc:1428] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:14.151321 17495 ts_tablet_manager.cc:1397] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 07:16:14.151727 17495 raft_consensus.cc:348] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.151834 17495 raft_consensus.cc:374] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:14.151866 17495 raft_consensus.cc:729] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7051405a739d4851a5900f1f89c6d3d5, State: Initialized, Role: FOLLOWER
I20260924 07:16:14.151980 17495 consensus_queue.cc:260] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.152212 17495 ts_tablet_manager.cc:1428] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:14.152237 17494 tablet_bootstrap.cc:492] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: No bootstrap required, opened a new log
I20260924 07:16:14.152323 17494 ts_tablet_manager.cc:1397] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260924 07:16:14.152779 17494 raft_consensus.cc:348] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.152891 17494 raft_consensus.cc:374] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:14.152922 17494 raft_consensus.cc:729] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39720af34ec43c5afe592d9dce9b6c0, State: Initialized, Role: FOLLOWER
I20260924 07:16:14.153568 17494 consensus_queue.cc:260] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.154253 17494 ts_tablet_manager.cc:1428] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:16:14.194785 17515 raft_consensus.cc:482] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:14.194933 17515 raft_consensus.cc:504] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.195477 17515 leader_election.cc:288] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135), 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765)
I20260924 07:16:14.195930 17145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d74337d664a74a08a676620b1a0f2" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39720af34ec43c5afe592d9dce9b6c0" is_pre_election: true
I20260924 07:16:14.195928 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d74337d664a74a08a676620b1a0f2" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7051405a739d4851a5900f1f89c6d3d5" is_pre_election: true
I20260924 07:16:14.196094 17425 raft_consensus.cc:2454] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 0.
I20260924 07:16:14.196089 17145 raft_consensus.cc:2454] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 0.
I20260924 07:16:14.196380 17216 leader_election.cc:302] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81654a93e916468a978ee4d1a221ab2b, e39720af34ec43c5afe592d9dce9b6c0; no voters:
I20260924 07:16:14.196570 17515 raft_consensus.cc:2781] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:14.196650 17515 raft_consensus.cc:482] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:14.196699 17515 raft_consensus.cc:3037] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.198040 17515 raft_consensus.cc:504] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.198411 17515 leader_election.cc:288] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 election: Requested vote from peers e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135), 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765)
I20260924 07:16:14.198781 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d74337d664a74a08a676620b1a0f2" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7051405a739d4851a5900f1f89c6d3d5"
I20260924 07:16:14.198783 17145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99d74337d664a74a08a676620b1a0f2" candidate_uuid: "81654a93e916468a978ee4d1a221ab2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e39720af34ec43c5afe592d9dce9b6c0"
I20260924 07:16:14.198908 17145 raft_consensus.cc:3037] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.198913 17425 raft_consensus.cc:3037] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.199940 17425 raft_consensus.cc:2454] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 1.
I20260924 07:16:14.200069 17145 raft_consensus.cc:2454] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81654a93e916468a978ee4d1a221ab2b in term 1.
I20260924 07:16:14.200280 17219 leader_election.cc:302] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7051405a739d4851a5900f1f89c6d3d5, 81654a93e916468a978ee4d1a221ab2b; no voters:
I20260924 07:16:14.200466 17515 raft_consensus.cc:2781] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:14.200558 17515 raft_consensus.cc:686] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 1 LEADER]: Becoming Leader. State: Replica: 81654a93e916468a978ee4d1a221ab2b, State: Running, Role: LEADER
I20260924 07:16:14.200704 17515 consensus_queue.cc:237] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.202491 16987 catalog_manager.cc:5495] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 81654a93e916468a978ee4d1a221ab2b (127.15.127.130). New cstate: current_term: 1 leader_uuid: "81654a93e916468a978ee4d1a221ab2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:16:14 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:14.244112 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.234637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34729 (local address 127.0.0.1:51198)
0924 07:16:14.234960 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.234978 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.235088 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.235373 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.235376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.235770 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:14.235871 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.235877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.237184 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.237187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.237843 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.237851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.238400 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.238411 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.238473 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.239079 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:14.239091 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:14.240528 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:14.242627 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:14.242635 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:14.242639 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:14.242860 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:14.243177 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:14.243179 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:14.243181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:14.243253 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:14.243681 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:14.243687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:14.243781 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.243867 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.243972 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":115,"threads_started":1}
I20260924 07:16:14.244268 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.234756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51198 (local address 127.15.127.190:34729)
0924 07:16:14.234904 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.234909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.235060 (+ 151us) server_negotiation.cc:407] Connection header received
0924 07:16:14.235183 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.235193 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.235247 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.235344 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:14.236066 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.237072 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.237932 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.238276 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.238381 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.240666 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:14.240670 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:14.240672 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:14.240699 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:14.242511 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:14.242963 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:14.242968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:14.242972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:14.243077 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:14.243339 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:14.243344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:14.243345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:14.243580 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:14.243679 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.243950 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.244089 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:16:14.248494 16987 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:51198:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:16:14.249353 16987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:14.254586 17125 tablet_service.cc:1467] Processing CreateTablet for tablet 9af4fef6c5f648d9b3e1137d249e1212 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c46a17e8426547f4a2cac7faa4288ca7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:14.254566 17265 tablet_service.cc:1467] Processing CreateTablet for tablet 9af4fef6c5f648d9b3e1137d249e1212 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c46a17e8426547f4a2cac7faa4288ca7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:14.254978 17125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af4fef6c5f648d9b3e1137d249e1212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:14.254978 17265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af4fef6c5f648d9b3e1137d249e1212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:14.257906 17494 tablet_bootstrap.cc:492] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: Bootstrap starting.
I20260924 07:16:14.258168 17405 tablet_service.cc:1467] Processing CreateTablet for tablet 9af4fef6c5f648d9b3e1137d249e1212 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c46a17e8426547f4a2cac7faa4288ca7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:14.258567 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9af4fef6c5f648d9b3e1137d249e1212. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:14.258729 17494 tablet_bootstrap.cc:654] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:14.259860 17494 tablet_bootstrap.cc:492] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: No bootstrap required, opened a new log
I20260924 07:16:14.259922 17494 ts_tablet_manager.cc:1397] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:16:14.260255 17494 raft_consensus.cc:348] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.260339 17494 raft_consensus.cc:374] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:14.260360 17494 raft_consensus.cc:729] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e39720af34ec43c5afe592d9dce9b6c0, State: Initialized, Role: FOLLOWER
I20260924 07:16:14.260437 17494 consensus_queue.cc:260] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.260705 17494 ts_tablet_manager.cc:1428] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:16:14.260658 17501 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.260780 17501 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.261092 17496 tablet_bootstrap.cc:492] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: Bootstrap starting.
I20260924 07:16:14.262138 17496 tablet_bootstrap.cc:654] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:14.262205 17495 tablet_bootstrap.cc:492] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: Bootstrap starting.
I20260924 07:16:14.263226 17495 tablet_bootstrap.cc:654] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:14.263760 17496 tablet_bootstrap.cc:492] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: No bootstrap required, opened a new log
I20260924 07:16:14.263852 17496 ts_tablet_manager.cc:1397] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:14.264334 17496 raft_consensus.cc:348] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.264436 17496 raft_consensus.cc:374] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:14.264489 17496 raft_consensus.cc:729] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81654a93e916468a978ee4d1a221ab2b, State: Initialized, Role: FOLLOWER
I20260924 07:16:14.264639 17496 consensus_queue.cc:260] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.264953 17496 ts_tablet_manager.cc:1428] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 07:16:14.264959 17515 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.265218 17515 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.265278 17495 tablet_bootstrap.cc:492] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: No bootstrap required, opened a new log
I20260924 07:16:14.265353 17495 ts_tablet_manager.cc:1397] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:16:14.265765 17495 raft_consensus.cc:348] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.265889 17495 raft_consensus.cc:374] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:14.265928 17495 raft_consensus.cc:729] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7051405a739d4851a5900f1f89c6d3d5, State: Initialized, Role: FOLLOWER
I20260924 07:16:14.266064 17495 consensus_queue.cc:260] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.266395 17495 ts_tablet_manager.cc:1428] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:16:14.266424 17502 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.266582 17502 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.434423 17501 raft_consensus.cc:482] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:14.434594 17501 raft_consensus.cc:504] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.435738 17501 leader_election.cc:288] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81654a93e916468a978ee4d1a221ab2b (127.15.127.130:38565), 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765)
I20260924 07:16:14.440179 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.436548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:59069 (local address 127.15.127.131:38765)
0924 07:16:14.436691 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.436696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.436816 (+ 120us) server_negotiation.cc:407] Connection header received
0924 07:16:14.436983 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.436987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.437060 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.437154 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:14.437432 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.438012 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.439145 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.439542 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.439676 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.439729 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.439968 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.440052 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:16:14.440189 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.436093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:45823 (local address 127.15.127.130:38565)
0924 07:16:14.436256 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.436262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.436473 (+ 211us) server_negotiation.cc:407] Connection header received
0924 07:16:14.436614 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.436618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.436698 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.436810 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:14.437140 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.437755 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.439143 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.439551 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.439686 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.439743 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.439954 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.440044 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:16:14.440215 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.436416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38765 (local address 127.15.127.129:59069)
0924 07:16:14.436737 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.436747 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.436882 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.437177 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.437179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.437215 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:14.437310 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.437314 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.438136 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.438139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.439023 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.439032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.439662 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.439665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.439809 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.439849 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.439938 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.440073 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":105,"spinlock_wait_cycles":6016,"thread_start_us":94,"threads_started":1}
I20260924 07:16:14.440240 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.435992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38565 (local address 127.15.127.129:45823)
0924 07:16:14.436333 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.436346 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.436508 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.436821 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.436824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.436870 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:14.436982 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.436987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.437888 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.437891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.439022 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.439031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.439690 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.439693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.439804 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.439848 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.439936 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.440052 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":37,"spinlock_wait_cycles":25728,"thread_start_us":126,"threads_started":1}
I20260924 07:16:14.440893 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af4fef6c5f648d9b3e1137d249e1212" candidate_uuid: "e39720af34ec43c5afe592d9dce9b6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7051405a739d4851a5900f1f89c6d3d5" is_pre_election: true
I20260924 07:16:14.440951 17285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af4fef6c5f648d9b3e1137d249e1212" candidate_uuid: "e39720af34ec43c5afe592d9dce9b6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81654a93e916468a978ee4d1a221ab2b" is_pre_election: true
I20260924 07:16:14.441073 17425 raft_consensus.cc:2454] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e39720af34ec43c5afe592d9dce9b6c0 in term 0.
I20260924 07:16:14.441232 17285 raft_consensus.cc:2454] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e39720af34ec43c5afe592d9dce9b6c0 in term 0.
I20260924 07:16:14.441488 17079 leader_election.cc:302] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7051405a739d4851a5900f1f89c6d3d5, e39720af34ec43c5afe592d9dce9b6c0; no voters:
I20260924 07:16:14.441772 17501 raft_consensus.cc:2781] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:14.441850 17501 raft_consensus.cc:482] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:14.441900 17501 raft_consensus.cc:3037] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.442888 17501 raft_consensus.cc:504] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.443233 17501 leader_election.cc:288] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [CANDIDATE]: Term 1 election: Requested vote from peers 81654a93e916468a978ee4d1a221ab2b (127.15.127.130:38565), 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765)
I20260924 07:16:14.443557 17285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af4fef6c5f648d9b3e1137d249e1212" candidate_uuid: "e39720af34ec43c5afe592d9dce9b6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81654a93e916468a978ee4d1a221ab2b"
I20260924 07:16:14.443670 17285 raft_consensus.cc:3037] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.443696 17425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9af4fef6c5f648d9b3e1137d249e1212" candidate_uuid: "e39720af34ec43c5afe592d9dce9b6c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7051405a739d4851a5900f1f89c6d3d5"
I20260924 07:16:14.443809 17425 raft_consensus.cc:3037] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:14.444634 17285 raft_consensus.cc:2454] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e39720af34ec43c5afe592d9dce9b6c0 in term 1.
I20260924 07:16:14.444761 17425 raft_consensus.cc:2454] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e39720af34ec43c5afe592d9dce9b6c0 in term 1.
I20260924 07:16:14.444900 17079 leader_election.cc:302] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81654a93e916468a978ee4d1a221ab2b, e39720af34ec43c5afe592d9dce9b6c0; no voters:
I20260924 07:16:14.445086 17501 raft_consensus.cc:2781] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:14.445286 17501 raft_consensus.cc:686] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 LEADER]: Becoming Leader. State: Replica: e39720af34ec43c5afe592d9dce9b6c0, State: Running, Role: LEADER
I20260924 07:16:14.445546 17501 consensus_queue.cc:237] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } }
I20260924 07:16:14.446949 17536 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: TxnStatusTablet state changed. Reason: New leader e39720af34ec43c5afe592d9dce9b6c0. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.447053 17536 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:14.447306 17538 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:16:14.447479 17538 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:16:14.447894 16987 catalog_manager.cc:5495] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129). New cstate: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } health_report { overall_health: UNKNOWN } } }
I20260924 07:16:14.503872 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.499757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45135 (local address 127.0.0.1:58140)
0924 07:16:14.499937 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.499952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.500058 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.500375 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.500378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.500386 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:14.500498 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.500504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.501322 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.501326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.502289 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.502301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.502747 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.502751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.502850 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.502893 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.503577 (+ 684us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.503593 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.503663 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.503757 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 07:16:14.503868 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.499896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.15.127.129:45135)
0924 07:16:14.500027 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.500030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.500044 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:14.500205 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.500207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.500260 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.500341 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:14.500599 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.501218 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.502401 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.502623 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.502735 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.503045 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.503479 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.503607 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.503672 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.503726 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:16:14.506729 17425 raft_consensus.cc:1261] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Refusing update from remote peer e39720af34ec43c5afe592d9dce9b6c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:14.507001 17285 raft_consensus.cc:1261] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Refusing update from remote peer e39720af34ec43c5afe592d9dce9b6c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:14.507339 17501 consensus_queue.cc:1035] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:14.507534 17536 consensus_queue.cc:1035] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:14.509086 17502 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: TxnStatusTablet state changed. Reason: New leader e39720af34ec43c5afe592d9dce9b6c0. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.509217 17502 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.510803 17536 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.510919 17536 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:14.513247 17502 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.513353 17502 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P 7051405a739d4851a5900f1f89c6d3d5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.514991 17501 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.515094 17501 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P e39720af34ec43c5afe592d9dce9b6c0: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:14.515085 17515 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: TxnStatusTablet state changed. Reason: New leader e39720af34ec43c5afe592d9dce9b6c0. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.515208 17515 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.516072 17508 mvcc.cc:205] Tried to move back new op lower bound from 7332799178774691840 to 7332799178531651584. Current Snapshot: MvccSnapshot[applied={T|T < 7332799178774691840}]
I20260924 07:16:14.517149 17515 tablet_replica.cc:439] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e39720af34ec43c5afe592d9dce9b6c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 } } peers { permanent_uuid: "81654a93e916468a978ee4d1a221ab2b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38565 } } peers { permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 } } }
I20260924 07:16:14.517259 17515 tablet_replica.cc:442] T 9af4fef6c5f648d9b3e1137d249e1212 P 81654a93e916468a978ee4d1a221ab2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:14.527529 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.523798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:52521 (local address 127.15.127.129:45135)
0924 07:16:14.523948 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.523954 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.524163 (+ 209us) server_negotiation.cc:407] Connection header received
0924 07:16:14.524255 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.524257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.524300 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.524362 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:14.524626 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.525239 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.526477 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.526695 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.526775 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.527091 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.527203 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.527259 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.527378 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.527423 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:16:14.527599 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.523716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45135 (local address 127.15.127.130:52521)
0924 07:16:14.524047 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.524062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.524169 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.524392 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.524395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.524420 (+ 25us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:14.524532 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.524538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.525342 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.525346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.526357 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.526367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.526807 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.526810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.526900 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.526953 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.527289 (+ 336us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.527295 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.527361 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.527448 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":128,"threads_started":1}
I20260924 07:16:14.534349 17425 raft_consensus.cc:1261] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Refusing update from remote peer 81654a93e916468a978ee4d1a221ab2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:14.534349 17145 raft_consensus.cc:1261] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Refusing update from remote peer 81654a93e916468a978ee4d1a221ab2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:14.534756 17505 consensus_queue.cc:1035] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e39720af34ec43c5afe592d9dce9b6c0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:14.534883 17515 consensus_queue.cc:1035] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7051405a739d4851a5900f1f89c6d3d5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:14.564689 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.555896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:55245 (local address 127.15.127.130:38565)
0924 07:16:14.556424 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.556429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.561341 (+ 4912us) server_negotiation.cc:407] Connection header received
0924 07:16:14.561464 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.561467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.561511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.561595 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:14.561891 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.562412 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.563370 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.563699 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.563819 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.564165 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.564271 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.564338 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.564497 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.564549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444}
I20260924 07:16:14.564739 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.555543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38565 (local address 127.15.127.129:55245)
0924 07:16:14.561239 (+ 5696us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:14.561256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:14.561372 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:14.561607 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:14.561611 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:14.561619 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:14.561786 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.561792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.562538 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.562542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.563260 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:14.563273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.563829 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.563833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:14.563931 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.564004 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:14.564380 (+ 376us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:14.564389 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:16:14.564481 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:16:14.564561 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5541,"thread_start_us":116,"threads_started":1}
I20260924 07:16:14.591446 16988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:14.591593 16988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:14.593678 16988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf5f47ecd05a4285845a6544792eb6d0: Sending DeleteTablet for 3 replicas of tablet f99d74337d664a74a08a676620b1a0f2
I20260924 07:16:14.594226 17125 tablet_service.cc:1514] Processing DeleteTablet for tablet f99d74337d664a74a08a676620b1a0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:14 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:58126
I20260924 07:16:14.594303 17265 tablet_service.cc:1514] Processing DeleteTablet for tablet f99d74337d664a74a08a676620b1a0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:14 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40868
I20260924 07:16:14.594388 17405 tablet_service.cc:1514] Processing DeleteTablet for tablet f99d74337d664a74a08a676620b1a0f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:14 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:48206
I20260924 07:16:14.594434 17518 tablet_replica.cc:330] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: stopping tablet replica
I20260924 07:16:14.594489 17519 tablet_replica.cc:330] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: stopping tablet replica
I20260924 07:16:14.594555 17520 tablet_replica.cc:330] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: stopping tablet replica
I20260924 07:16:14.594550 17518 raft_consensus.cc:2229] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:14.594599 17519 raft_consensus.cc:2229] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:14.594677 17518 raft_consensus.cc:2258] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:14.594661 17520 raft_consensus.cc:2229] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:14.594784 17520 raft_consensus.cc:2258] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:14.594784 17519 raft_consensus.cc:2258] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:14.595561 17519 ts_tablet_manager.cc:1905] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:14.595564 17520 ts_tablet_manager.cc:1905] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:14.595597 17518 ts_tablet_manager.cc:1905] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:14.598138 17520 ts_tablet_manager.cc:1918] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:14.598197 17520 log.cc:1196] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/f99d74337d664a74a08a676620b1a0f2
I20260924 07:16:14.598379 17520 ts_tablet_manager.cc:1939] T f99d74337d664a74a08a676620b1a0f2 P 7051405a739d4851a5900f1f89c6d3d5: Deleting consensus metadata
I20260924 07:16:14.598362 17519 ts_tablet_manager.cc:1918] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:14.598362 17518 ts_tablet_manager.cc:1918] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:14.598430 17519 log.cc:1196] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/f99d74337d664a74a08a676620b1a0f2
I20260924 07:16:14.598430 17518 log.cc:1196] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/f99d74337d664a74a08a676620b1a0f2
I20260924 07:16:14.598686 17519 ts_tablet_manager.cc:1939] T f99d74337d664a74a08a676620b1a0f2 P 81654a93e916468a978ee4d1a221ab2b: Deleting consensus metadata
I20260924 07:16:14.598742 17518 ts_tablet_manager.cc:1939] T f99d74337d664a74a08a676620b1a0f2 P e39720af34ec43c5afe592d9dce9b6c0: Deleting consensus metadata
I20260924 07:16:14.599332 16975 catalog_manager.cc:4841] TS 81654a93e916468a978ee4d1a221ab2b (127.15.127.130:38565): tablet f99d74337d664a74a08a676620b1a0f2 (table test-table [id=7a0ec225e1f94b819ee8d114f12d11df]) successfully deleted
I20260924 07:16:14.599332 16972 catalog_manager.cc:4841] TS e39720af34ec43c5afe592d9dce9b6c0 (127.15.127.129:45135): tablet f99d74337d664a74a08a676620b1a0f2 (table test-table [id=7a0ec225e1f94b819ee8d114f12d11df]) successfully deleted
I20260924 07:16:14.599522 16975 catalog_manager.cc:4841] TS 7051405a739d4851a5900f1f89c6d3d5 (127.15.127.131:38765): tablet f99d74337d664a74a08a676620b1a0f2 (table test-table [id=7a0ec225e1f94b819ee8d114f12d11df]) successfully deleted
Sep 24 07:16:14 dist-test-slave-zx46 krb5kdc[16943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234173, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:14.612586 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.596102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.15.127.129:45135)
0924 07:16:14.596238 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.596243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.598785 (+ 2542us) server_negotiation.cc:407] Connection header received
0924 07:16:14.598979 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.598982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.599032 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.599083 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:14.600517 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.601075 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.601968 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.602206 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.602317 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.608891 (+ 6574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:14.608901 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:14.608916 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:14.608945 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:14.610715 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:14.611215 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:14.611224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:14.611230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:14.611308 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:14.611584 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:14.611589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:14.611591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:14.611910 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:14.612128 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.612292 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.612415 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:16:14.613158 17185 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58144
I20260924 07:16:14.622710 17037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:14.614871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51206 (local address 127.15.127.190:34729)
0924 07:16:14.614998 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:14.615003 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:14.615135 (+ 132us) server_negotiation.cc:407] Connection header received
0924 07:16:14.615234 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:14.615237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:14.615280 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:14.615346 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:14.616085 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.616605 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.617777 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:14.617991 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:14.618068 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:14.619507 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:14.619510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:14.619512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:14.619534 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:14.620965 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:14.621485 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:14.621487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:14.621488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:14.621558 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:14.621820 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:14.621822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:14.621823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:14.621999 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:14.622095 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:14.622402 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 07:16:14.622495 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 07:16:14.623108 16988 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51206
I20260924 07:16:14.624264 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17060
I20260924 07:16:14.632525 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17200
I20260924 07:16:14.640807 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17340
I20260924 07:16:14.646494 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 16956
2026-09-24T07:16:14Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2930 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:14 dist-test-slave-zx46 krb5kdc[17559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:14 dist-test-slave-zx46 krb5kdc[17559](info): set up 2 sockets
Sep 24 07:16:14 dist-test-slave-zx46 krb5kdc[17559](info): commencing operation
krb5kdc: starting...
W20260924 07:16:15.693246 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:15 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:15.819882 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:34489
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:15.928341 17572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:15.928658 17572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:15.928740 17572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:15.932665 17572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:15.932770 17572 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:16:15.932802 17572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:15.932847 17572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:15.932883 17572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:15.932919 17572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:15.938135 17572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:34489
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:15.939216 17572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:15.940240 17572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:15.948555 17580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:15.948575 17578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:15.948571 17577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:15.948745 17572 server_base.cc:1034] running on GCE node
I20260924 07:16:15.949437 17572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:15.950526 17572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:15.951711 17572 hybrid_clock.cc:648] HybridClock initialized: now 1790234175951691 us; error 30 us; skew 500 ppm
Sep 24 07:16:15 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234175, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:15.954512 17572 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:15.955371 17572 webserver.cc:460] Webserver started at http://127.15.127.190:40895/ using document root <none> and password file <none>
I20260924 07:16:15.955926 17572 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:15.956012 17572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:15.956240 17572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:15.958324 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "5610b6156c604c189791275d4dcec374"
format_stamp: "Formatted at 2026-09-24 07:16:15 on dist-test-slave-zx46"
server_key: "8e832ff3ad33733238393f185bcd6899"
server_key_iv: "9d42c55db61f311e3b563360eda810e1"
server_key_version: "encryptionkey@0"
I20260924 07:16:15.958896 17572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "5610b6156c604c189791275d4dcec374"
format_stamp: "Formatted at 2026-09-24 07:16:15 on dist-test-slave-zx46"
server_key: "8e832ff3ad33733238393f185bcd6899"
server_key_iv: "9d42c55db61f311e3b563360eda810e1"
server_key_version: "encryptionkey@0"
I20260924 07:16:15.962868 17572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:15.965350 17587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:15.966583 17572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:16:15.966725 17572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "5610b6156c604c189791275d4dcec374"
format_stamp: "Formatted at 2026-09-24 07:16:15 on dist-test-slave-zx46"
server_key: "8e832ff3ad33733238393f185bcd6899"
server_key_iv: "9d42c55db61f311e3b563360eda810e1"
server_key_version: "encryptionkey@0"
I20260924 07:16:15.966835 17572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:16.004303 17572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:16.007805 17572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:16.008026 17572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:16.015853 17572 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:34489
I20260924 07:16:16.015971 17649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:34489 every 8 connection(s)
I20260924 07:16:16.016907 17572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:16.023103 17650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:16.026582 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17572
I20260924 07:16:16.026671 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:16.026922 15870 external_mini_cluster.cc:1428] Setting key a4a905d9871959181213153271e742b3
I20260924 07:16:16.029556 17650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Bootstrap starting.
I20260924 07:16:16.032222 17650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:16.033084 17650 log.cc:824] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:16.035094 17650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: No bootstrap required, opened a new log
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:16.039482 17650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5610b6156c604c189791275d4dcec374" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34489 } }
I20260924 07:16:16.039772 17650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:16.039834 17650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5610b6156c604c189791275d4dcec374, State: Initialized, Role: FOLLOWER
I20260924 07:16:16.040336 17650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5610b6156c604c189791275d4dcec374" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34489 } }
I20260924 07:16:16.040493 17650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:16.040555 17650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:16.040638 17650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:16.041709 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.028322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.15.127.190:34489)
0924 07:16:16.028743 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.028763 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.028806 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:16:16.029628 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.029653 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.030059 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.030420 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.031034 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.032505 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.033446 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.034045 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.034206 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.036594 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.036601 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.036617 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.036649 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.038624 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.039017 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.039023 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.039029 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.039173 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.039414 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.039417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.039419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.039921 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.040119 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.040396 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.040699 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":111,"threads_started":1}
I20260924 07:16:16.041731 17650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5610b6156c604c189791275d4dcec374" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34489 } }
I20260924 07:16:16.042069 17650 leader_election.cc:302] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5610b6156c604c189791275d4dcec374; no voters:
I20260924 07:16:16.042379 17650 leader_election.cc:288] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:16.042490 17655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:16.042698 17655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [term 1 LEADER]: Becoming Leader. State: Replica: 5610b6156c604c189791275d4dcec374, State: Running, Role: LEADER
I20260924 07:16:16.043015 17655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5610b6156c604c189791275d4dcec374" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34489 } }
I20260924 07:16:16.043584 17650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:16.044526 17657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5610b6156c604c189791275d4dcec374. Latest consensus state: current_term: 1 leader_uuid: "5610b6156c604c189791275d4dcec374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5610b6156c604c189791275d4dcec374" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34489 } } }
I20260924 07:16:16.044658 17657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:16.045045 17656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5610b6156c604c189791275d4dcec374" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5610b6156c604c189791275d4dcec374" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34489 } } }
I20260924 07:16:16.045209 17656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:16.045378 17664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:16.048445 17664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:16.053777 17664 catalog_manager.cc:1347] Generated new cluster ID: 55a68a2a14fb412fa5b45b8eb294e807
I20260924 07:16:16.053840 17664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:16.079252 17664 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:16.080588 17664 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:16.087782 17664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Generated new TSK 0
I20260924 07:16:16.088639 17664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:16.119563 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34489
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:16.234306 17676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:16.234571 17676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:16.234619 17676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:16.238685 17676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:16.238754 17676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:16.238795 17676 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:16:16.238843 17676 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:16.238898 17676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:16.244310 17676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:16.245565 17676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:16.246667 17676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:16.256398 17681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:16.256381 17684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:16.256389 17682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:16.256690 17676 server_base.cc:1034] running on GCE node
I20260924 07:16:16.257294 17676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:16.257956 17676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:16.259151 17676 hybrid_clock.cc:648] HybridClock initialized: now 1790234176259132 us; error 35 us; skew 500 ppm
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:16.262210 17676 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:16.263322 17676 webserver.cc:460] Webserver started at http://127.15.127.129:40701/ using document root <none> and password file <none>
I20260924 07:16:16.264074 17676 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:16.264173 17676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:16.264433 17676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:16.267534 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "105d2904d94c47718377372b1d68a9c6"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "a90422d944f74b844a22b74feff7fb0f"
server_key_iv: "558cc322809c8394102ffc7179c504fe"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.268363 17676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "105d2904d94c47718377372b1d68a9c6"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "a90422d944f74b844a22b74feff7fb0f"
server_key_iv: "558cc322809c8394102ffc7179c504fe"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.274168 17676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 07:16:16.277782 17691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:16.279621 17676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:16.279778 17676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "105d2904d94c47718377372b1d68a9c6"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "a90422d944f74b844a22b74feff7fb0f"
server_key_iv: "558cc322809c8394102ffc7179c504fe"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.279942 17676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:16.303047 17676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:16.306847 17676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:16.307067 17676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:16.308674 17676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:16.308746 17676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:16.308820 17676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:16.308853 17676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:16.319598 17676 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:32841
I20260924 07:16:16.319720 17806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:32841 every 8 connection(s)
I20260924 07:16:16.320508 17676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:16.323343 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.309364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:47639 (local address 127.15.127.190:34489)
0924 07:16:16.309487 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.309498 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.311138 (+ 1640us) server_negotiation.cc:407] Connection header received
0924 07:16:16.312271 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.312274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.312333 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.312420 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.314092 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.314730 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.316547 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.316771 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.316849 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.319921 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.319924 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.319926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.319950 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.321687 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.322119 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.322122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.322123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.322192 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.322469 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.322471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.322473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.322645 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.322717 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.323129 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.323226 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:16:16.324115 17700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.310457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.15.127.129:47639)
0924 07:16:16.310977 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.311035 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.312064 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.312603 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.312612 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.313183 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:16.313917 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.313938 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.314855 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.314860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.316410 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.316420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.316935 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.316942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.317208 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.317890 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:16.317916 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:16.319728 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:16.321781 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.321786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.321796 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.322032 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.322268 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.322270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.322271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.322393 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.322728 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:16.322732 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:16.322939 (+ 207us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.323085 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.323324 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":127,"threads_started":1}
I20260924 07:16:16.325379 17807 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34489
I20260924 07:16:16.325513 17807 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:16.326038 17807 heartbeater.cc:511] Master 127.15.127.190:34489 requested a full tablet report, sending...
I20260924 07:16:16.326535 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17676
I20260924 07:16:16.326654 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:16.326946 15870 external_mini_cluster.cc:1428] Setting key 832e08f36edd61ae60089d65c5ddd125
I20260924 07:16:16.327808 17604 ts_manager.cc:194] Registered new tserver with Master: 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841)
I20260924 07:16:16.329377 17604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:47639
I20260924 07:16:16.337855 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.329789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.15.127.129:48397)
0924 07:16:16.330071 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.330087 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.330193 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.330513 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.330515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.330761 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:16.330903 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.330913 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.331912 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.331918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.332946 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.332959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.333436 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.333442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.333571 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.334054 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:16.334066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:16.334420 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:16.336425 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.336427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.336429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.336622 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.336908 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.336910 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.336911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.336979 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.337444 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:16.337446 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:16.337515 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.337615 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.337711 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":107,"threads_started":1}
I20260924 07:16:16.337994 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.330182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:48397 (local address 127.15.127.190:34489)
0924 07:16:16.330312 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.330318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.330331 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:16.330369 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.330371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.330404 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.330493 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.331040 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.331772 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.333116 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.333325 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.333403 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.334559 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.334566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.334571 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.334604 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.336310 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.336718 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.336722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.336725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.336815 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.337104 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.337108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.337110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.337347 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.337459 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.337721 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.337837 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:16.357223 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34489
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:16.468879 17816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:16.469223 17816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:16.469314 17816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:16.473654 17816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:16.473713 17816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:16.473754 17816 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:16:16.473807 17816 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:16.473865 17816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:16.479183 17816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:16.480258 17816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:16.481427 17816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:16.492277 17822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:16.492278 17824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:16.492326 17821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:16.492547 17816 server_base.cc:1034] running on GCE node
I20260924 07:16:16.493181 17816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:16.493840 17816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:16.495000 17816 hybrid_clock.cc:648] HybridClock initialized: now 1790234176494984 us; error 30 us; skew 500 ppm
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:16.497751 17816 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:16.498850 17816 webserver.cc:460] Webserver started at http://127.15.127.130:44629/ using document root <none> and password file <none>
I20260924 07:16:16.499540 17816 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:16.499609 17816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:16.499827 17816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:16.501945 17816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "5e4b1d96da3d4301bada9736ea36e421"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "99751ea9ebb24bf539e1b65b8957a620"
server_key_iv: "9fbb4d70305b74d0a5a8c4563aa14d9d"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.502525 17816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "5e4b1d96da3d4301bada9736ea36e421"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "99751ea9ebb24bf539e1b65b8957a620"
server_key_iv: "9fbb4d70305b74d0a5a8c4563aa14d9d"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.506232 17816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:16.508569 17831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:16.509902 17816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:16.510062 17816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "5e4b1d96da3d4301bada9736ea36e421"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "99751ea9ebb24bf539e1b65b8957a620"
server_key_iv: "9fbb4d70305b74d0a5a8c4563aa14d9d"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.510181 17816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:16.530417 17816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:16.533669 17816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:16.533903 17816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:16.535517 17816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:16.535595 17816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:16.535674 17816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:16.535712 17816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:16.548513 17816 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:42735
I20260924 07:16:16.548529 17946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:42735 every 8 connection(s)
I20260924 07:16:16.549850 17816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:16.551605 17840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.536892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.15.127.130:43193)
0924 07:16:16.537412 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.537455 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.538351 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.538994 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.539024 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.539385 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:16.539858 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.539872 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.540709 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.540712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.541767 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.541777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.542549 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.542556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.542826 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.543511 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:16.543533 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:16.545963 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:16.548337 (+ 2374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.548343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.548361 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.548808 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.549208 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.549211 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.549212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.549370 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.549885 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:16.549890 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:16.550292 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.550499 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.550849 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":140,"threads_started":1}
I20260924 07:16:16.552242 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.536277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:43193 (local address 127.15.127.190:34489)
0924 07:16:16.536407 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.536410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.537561 (+ 1151us) server_negotiation.cc:407] Connection header received
0924 07:16:16.538556 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.538559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.538616 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.538713 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.540007 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.540537 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.541884 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.542353 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.542529 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.546150 (+ 3621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.546155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.546162 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.546203 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.548170 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.548947 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.548951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.548953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.549091 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.549497 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.549501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.549503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.549770 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.549888 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.552032 (+ 2144us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.552136 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:16.553369 17947 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34489
I20260924 07:16:16.553509 17947 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:16.553653 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17816
I20260924 07:16:16.553766 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:16.554047 15870 external_mini_cluster.cc:1428] Setting key b35f3483c19861df13cb9c71a37d8c0a
I20260924 07:16:16.554054 17947 heartbeater.cc:511] Master 127.15.127.190:34489 requested a full tablet report, sending...
I20260924 07:16:16.555299 17604 ts_manager.cc:194] Registered new tserver with Master: 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:16.556129 17604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:43193
I20260924 07:16:16.564129 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.555740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:49915 (local address 127.15.127.190:34489)
0924 07:16:16.555862 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.555865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.556742 (+ 877us) server_negotiation.cc:407] Connection header received
0924 07:16:16.556877 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.556881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.556928 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.557037 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.557592 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.558437 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.559400 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.559634 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.559749 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.560856 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.560860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.560863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.560885 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.562470 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.562899 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.562904 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.562906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.562989 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.563265 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.563270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.563272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.563485 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.563594 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.563845 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.563975 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:16:16.564152 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.555673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.15.127.130:49915)
0924 07:16:16.556663 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.556680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.556781 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.557077 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.557080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.557314 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:16.557451 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.557459 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.558577 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.558581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.559281 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.559289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.559754 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.559757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.559846 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.560393 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:16.560413 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:16.560722 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:16.562584 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.562588 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.562591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.562797 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.563079 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.563084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.563086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.563176 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.563584 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:16.563586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:16.563684 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.563792 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.563952 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:16.584455 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34489
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:16.698976 17956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:16.699290 17956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:16.699383 17956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:16.703557 17956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:16.703646 17956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:16.703718 17956 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:16:16.703761 17956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:16.703831 17956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:16.709328 17956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:16.710439 17956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:16.711494 17956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:16.721236 17962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:16.721216 17964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:16.721376 17956 server_base.cc:1034] running on GCE node
W20260924 07:16:16.721266 17961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:16.721994 17956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:16.722664 17956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:16.723867 17956 hybrid_clock.cc:648] HybridClock initialized: now 1790234176723850 us; error 30 us; skew 500 ppm
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:16.726889 17956 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:16.727777 17956 webserver.cc:460] Webserver started at http://127.15.127.131:45933/ using document root <none> and password file <none>
I20260924 07:16:16.728310 17956 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:16.728368 17956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:16.728540 17956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:16.730726 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "de520ec398174145aca35db4306d1ff6"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "850fb1ceb75044b14de7e14d93debc95"
server_key_iv: "0c8bd7bb91684678b6f6cb43b71918b3"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.731247 17956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "de520ec398174145aca35db4306d1ff6"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "850fb1ceb75044b14de7e14d93debc95"
server_key_iv: "0c8bd7bb91684678b6f6cb43b71918b3"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.735100 17956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:16.737506 17971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:16.738739 17956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:16.738834 17956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "de520ec398174145aca35db4306d1ff6"
format_stamp: "Formatted at 2026-09-24 07:16:16 on dist-test-slave-zx46"
server_key: "850fb1ceb75044b14de7e14d93debc95"
server_key_iv: "0c8bd7bb91684678b6f6cb43b71918b3"
server_key_version: "encryptionkey@0"
I20260924 07:16:16.738919 17956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:16.759835 17956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:16.762832 17956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:16.762989 17956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:16.764348 17956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:16.764432 17956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:16.764495 17956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:16.764547 17956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:16.775435 17956 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:38845
I20260924 07:16:16.775511 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:38845 every 8 connection(s)
I20260924 07:16:16.776369 17956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:16.778244 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.765217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:43411 (local address 127.15.127.190:34489)
0924 07:16:16.765379 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.765384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.766227 (+ 843us) server_negotiation.cc:407] Connection header received
0924 07:16:16.767303 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.767307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.767373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.767490 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.768675 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.769586 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.771111 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.771411 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.771541 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.774582 (+ 3041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.774586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.774589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.774624 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.776157 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.776772 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.776775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.776777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.776866 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.777272 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.777275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.777277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.777463 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.777543 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.778012 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.778110 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:16:16.779032 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.765626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.15.127.131:43411)
0924 07:16:16.766061 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.766109 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.767076 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.767687 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.767697 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.768047 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:16.768495 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.768507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.769745 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.769749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.770983 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.770993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.771591 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.771595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.771785 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.772357 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:16.772376 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:16.774418 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:16.776274 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.776280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.776294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.776649 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.776965 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.776968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.776970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.777158 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.777577 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:16.777582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:16.777799 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.777953 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.778193 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":119,"threads_started":1}
I20260924 07:16:16.780148 18087 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34489
I20260924 07:16:16.780313 18087 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:16.780827 18087 heartbeater.cc:511] Master 127.15.127.190:34489 requested a full tablet report, sending...
I20260924 07:16:16.781297 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 17956
I20260924 07:16:16.781400 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:16.781708 15870 external_mini_cluster.cc:1428] Setting key af259be49d7a6e9b67cdcb67b9f496bf
I20260924 07:16:16.781849 17604 ts_manager.cc:194] Registered new tserver with Master: de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845)
I20260924 07:16:16.782589 17604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:43411
I20260924 07:16:16.783751 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:16:16.791430 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.783369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:32935 (local address 127.15.127.190:34489)
0924 07:16:16.783466 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.783470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.783482 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:16.783519 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.783526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.783573 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.783646 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.784266 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.785083 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.786067 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.786334 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.786413 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.787788 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.787791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.787793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.787814 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.789583 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.790111 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.790115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.790116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.790206 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.790560 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.790563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.790564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.790748 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.790874 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.791162 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.791273 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:16:16.791450 18093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.783038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.15.127.131:32935)
0924 07:16:16.783308 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.783323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.783415 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.783747 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.783750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.783953 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:16.784140 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.784147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.785211 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.785214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.785952 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.785966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.786448 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.786454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.786568 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.787171 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:16.787191 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:16.787650 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:16.789701 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.789708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.789713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.789988 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.790310 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:16.790316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:16.790321 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:16.790427 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:16.790873 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:16.790876 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:16.790973 (+ 97us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.791098 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.791253 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":18,"thread_start_us":106,"threads_started":1}
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:16:16 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:16.811832 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.802941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.15.127.190:34489)
0924 07:16:16.803149 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.803153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.803167 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:16.803227 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.803229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.803268 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.803348 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:16.804048 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.804605 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.805568 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.805794 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.805881 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.808383 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:16.808387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:16.808389 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:16.808417 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:16.810402 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.810818 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.810821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.810822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.810896 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:16.811147 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:16.811149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:16.811151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:16.811325 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:16.811422 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.811613 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.811710 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260924 07:16:16.815011 17604 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:16.818187 17604 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:16.833508 18108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.828279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.15.127.130:42735)
0924 07:16:16.828827 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.828835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.828881 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:16:16.828951 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.828959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.829185 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.829347 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:16.829563 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.830559 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.831887 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.832718 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.832864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.833024 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.833246 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.833347 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":108,"threads_started":1}
I20260924 07:16:16.833495 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.827106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38845 (local address 127.0.0.1:39312)
0924 07:16:16.827490 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.827549 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.827706 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.828581 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.828587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.828608 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:16.828777 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.828785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.830262 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.830265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.831785 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.831796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.833106 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.833108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.833210 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.833234 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.833307 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.833372 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":95,"threads_started":1}
I20260924 07:16:16.833515 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.827552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42735 (local address 127.0.0.1:40008)
0924 07:16:16.828091 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.828105 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.828205 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.829320 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.829323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.829339 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:16.829442 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.829448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.830691 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.830697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.831733 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.831742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.832851 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.832855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.832971 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.833063 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.833205 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.833340 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260924 07:16:16.834052 18107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.827893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:32841 (local address 127.0.0.1:42972)
0924 07:16:16.829055 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:16.829071 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:16.829167 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:16.829672 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:16.829675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:16.829694 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:16.829776 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.829784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.831349 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.831353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.832332 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:16.832340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.833660 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.833666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:16.833776 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.833808 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:16:16.833884 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:16:16.833955 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"thread_start_us":87,"threads_started":1}
I20260924 07:16:16.834123 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.827294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.15.127.131:38845)
0924 07:16:16.827689 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.827701 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.827727 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:16:16.827930 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.827937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.828189 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.828382 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:16.828917 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.830135 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.831915 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.832718 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.832864 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.833024 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.833899 (+ 875us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.833993 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":99,"threads_started":1}
I20260924 07:16:16.834580 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:16.828509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42972 (local address 127.15.127.129:32841)
0924 07:16:16.828815 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:16.828825 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:16.829152 (+ 327us) server_negotiation.cc:407] Connection header received
0924 07:16:16.829292 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:16.829300 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:16.829512 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:16.829683 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:16.829881 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.831213 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.832465 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:16.833327 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:16.833449 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:16.833575 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:16.834317 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 07:16:16.834414 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":83,"threads_started":1}
I20260924 07:16:16.837100 17881 tablet_service.cc:1467] Processing CreateTablet for tablet 289f31e9afc74413bd3d2c9dbb98a355 (DEFAULT_TABLE table=test-table [id=5a3bbd2eef9546628cbf9ac9e0c313f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:16.837091 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 289f31e9afc74413bd3d2c9dbb98a355 (DEFAULT_TABLE table=test-table [id=5a3bbd2eef9546628cbf9ac9e0c313f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:16.837093 18021 tablet_service.cc:1467] Processing CreateTablet for tablet 289f31e9afc74413bd3d2c9dbb98a355 (DEFAULT_TABLE table=test-table [id=5a3bbd2eef9546628cbf9ac9e0c313f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:16.838214 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289f31e9afc74413bd3d2c9dbb98a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:16.838214 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289f31e9afc74413bd3d2c9dbb98a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:16.838253 17881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 289f31e9afc74413bd3d2c9dbb98a355. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:16.845762 18111 tablet_bootstrap.cc:492] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Bootstrap starting.
I20260924 07:16:16.846918 18112 tablet_bootstrap.cc:492] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Bootstrap starting.
I20260924 07:16:16.847206 18110 tablet_bootstrap.cc:492] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Bootstrap starting.
I20260924 07:16:16.848583 18111 tablet_bootstrap.cc:654] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:16.848882 18110 tablet_bootstrap.cc:654] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:16.849195 18112 tablet_bootstrap.cc:654] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:16.849658 18111 log.cc:824] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:16.849757 18110 log.cc:824] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:16.849987 18112 log.cc:824] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:16.851672 18111 tablet_bootstrap.cc:492] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: No bootstrap required, opened a new log
I20260924 07:16:16.851711 18110 tablet_bootstrap.cc:492] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: No bootstrap required, opened a new log
I20260924 07:16:16.851753 18112 tablet_bootstrap.cc:492] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: No bootstrap required, opened a new log
I20260924 07:16:16.851902 18112 ts_tablet_manager.cc:1397] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:16:16.851902 18110 ts_tablet_manager.cc:1397] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 07:16:16.851908 18111 ts_tablet_manager.cc:1397] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:16:16.855525 18111 raft_consensus.cc:348] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:16.855808 18111 raft_consensus.cc:374] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:16.855865 18111 raft_consensus.cc:729] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de520ec398174145aca35db4306d1ff6, State: Initialized, Role: FOLLOWER
I20260924 07:16:16.855926 18110 raft_consensus.cc:348] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:16.855926 18112 raft_consensus.cc:348] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:16.856191 18110 raft_consensus.cc:374] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:16.856191 18112 raft_consensus.cc:374] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:16.856242 18112 raft_consensus.cc:729] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Initialized, Role: FOLLOWER
I20260924 07:16:16.856242 18110 raft_consensus.cc:729] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105d2904d94c47718377372b1d68a9c6, State: Initialized, Role: FOLLOWER
I20260924 07:16:16.856372 18111 consensus_queue.cc:260] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:16.856662 18112 consensus_queue.cc:260] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:16.856659 18110 consensus_queue.cc:260] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:16.857594 18111 ts_tablet_manager.cc:1428] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:16:16.857610 18110 ts_tablet_manager.cc:1428] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 07:16:16.857610 18112 ts_tablet_manager.cc:1428] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:16:16.857978 18087 heartbeater.cc:503] Master 127.15.127.190:34489 was elected leader, sending a full tablet report...
I20260924 07:16:16.858812 17947 heartbeater.cc:503] Master 127.15.127.190:34489 was elected leader, sending a full tablet report...
I20260924 07:16:16.859697 17807 heartbeater.cc:503] Master 127.15.127.190:34489 was elected leader, sending a full tablet report...
W20260924 07:16:17.027326 18088 tablet.cc:2369] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:17.051163 17948 tablet.cc:2369] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:17.071997 17808 tablet.cc:2369] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:17.085348 18118 raft_consensus.cc:482] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:17.085594 18118 raft_consensus.cc:504] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:17.086899 18118 leader_election.cc:288] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845), 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841)
I20260924 07:16:17.091208 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.087310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:53785 (local address 127.15.127.131:38845)
0924 07:16:17.087462 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.087468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.087570 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:16:17.087768 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.087772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.087831 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.087935 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:17.088233 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.088873 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.090164 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.090594 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.090700 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.090769 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.090985 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.091068 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:16:17.091233 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.087601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:32841 (local address 127.15.127.130:58081)
0924 07:16:17.087869 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.087883 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.088010 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.088294 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.088296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.088317 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:17.088417 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.088423 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.089229 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.089231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.089978 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.090112 (+ 134us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.090811 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.090814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.090900 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.090917 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.091018 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.091083 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":10,"thread_start_us":98,"threads_started":1}
I20260924 07:16:17.091233 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.087743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:58081 (local address 127.15.127.129:32841)
0924 07:16:17.087871 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.087876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.087940 (+ 64us) server_negotiation.cc:407] Connection header received
0924 07:16:17.088128 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.088132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.088190 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.088269 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:17.088536 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.089119 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.090256 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.090684 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.090802 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.090850 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.091051 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.091132 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:16:17.091243 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.087171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38845 (local address 127.15.127.130:53785)
0924 07:16:17.087461 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.087479 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.087632 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.087961 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.087964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.087982 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:17.088119 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.088125 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.088984 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.088987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.089934 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.089943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.090730 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.090736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.090845 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.090878 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.090959 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.091052 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":22,"spinlock_wait_cycles":9472,"thread_start_us":83,"threads_started":1}
I20260924 07:16:17.091912 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289f31e9afc74413bd3d2c9dbb98a355" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6" is_pre_election: true
I20260924 07:16:17.092077 18036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289f31e9afc74413bd3d2c9dbb98a355" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de520ec398174145aca35db4306d1ff6" is_pre_election: true
I20260924 07:16:17.092271 17761 raft_consensus.cc:2454] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 0.
I20260924 07:16:17.092306 18036 raft_consensus.cc:2454] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 0.
I20260924 07:16:17.092805 17834 leader_election.cc:302] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4b1d96da3d4301bada9736ea36e421, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:17.093078 18118 raft_consensus.cc:2781] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:17.093190 18118 raft_consensus.cc:482] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:17.093253 18118 raft_consensus.cc:3037] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.094269 18118 raft_consensus.cc:504] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:17.094620 18118 leader_election.cc:288] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 election: Requested vote from peers de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845), 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841)
I20260924 07:16:17.094997 18036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289f31e9afc74413bd3d2c9dbb98a355" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de520ec398174145aca35db4306d1ff6"
I20260924 07:16:17.095086 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "289f31e9afc74413bd3d2c9dbb98a355" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6"
I20260924 07:16:17.095130 18036 raft_consensus.cc:3037] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.095216 17761 raft_consensus.cc:3037] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.096571 17761 raft_consensus.cc:2454] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 1.
I20260924 07:16:17.096764 18036 raft_consensus.cc:2454] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 1.
I20260924 07:16:17.096907 17834 leader_election.cc:302] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d2904d94c47718377372b1d68a9c6, 5e4b1d96da3d4301bada9736ea36e421; no voters:
I20260924 07:16:17.097103 18118 raft_consensus.cc:2781] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:17.097302 18118 raft_consensus.cc:686] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 LEADER]: Becoming Leader. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Running, Role: LEADER
I20260924 07:16:17.097654 18118 consensus_queue.cc:237] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } }
I20260924 07:16:17.101197 17603 catalog_manager.cc:5495] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130). New cstate: current_term: 1 leader_uuid: "5e4b1d96da3d4301bada9736ea36e421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } health_report { overall_health: HEALTHY } } }
Sep 24 07:16:17 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234175, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:17.147495 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.138860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34489 (local address 127.0.0.1:35972)
0924 07:16:17.139245 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.139258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.139400 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.139661 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.139664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.140106 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:17.140196 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.140203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.140943 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.140949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.141522 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.141530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.141974 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.141978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.142052 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.142597 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:17.142609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:17.144070 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:17.145983 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:17.145991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:17.145996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:17.146235 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:17.146580 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:17.146582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:17.146583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:17.146663 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:17.147000 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:17.147005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:17.147104 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.147208 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.147316 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":140,"threads_started":1}
I20260924 07:16:17.147493 17653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.138949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.15.127.190:34489)
0924 07:16:17.139173 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.139177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.139350 (+ 173us) server_negotiation.cc:407] Connection header received
0924 07:16:17.139504 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.139508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.139558 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.139645 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:17.140320 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.140835 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.141637 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.141846 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.141972 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.144195 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:17.144197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:17.144199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:17.144222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:17.145892 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:17.146387 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:17.146393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:17.146398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:17.146488 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:17.146742 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:17.146745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:17.146746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:17.146913 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:17.146988 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.147265 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.147368 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":95}
I20260924 07:16:17.151518 17603 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:35972:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:16:17.152333 17603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:17.157377 17881 tablet_service.cc:1467] Processing CreateTablet for tablet 92fbdbecdf364ad09664a791969f9c09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a4c4b3d01bf46d29a71a257ca4e9388]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:17.157346 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 92fbdbecdf364ad09664a791969f9c09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a4c4b3d01bf46d29a71a257ca4e9388]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:17.157598 18021 tablet_service.cc:1467] Processing CreateTablet for tablet 92fbdbecdf364ad09664a791969f9c09 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a4c4b3d01bf46d29a71a257ca4e9388]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:17.158063 17881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fbdbecdf364ad09664a791969f9c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:17.158062 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fbdbecdf364ad09664a791969f9c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:17.158102 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92fbdbecdf364ad09664a791969f9c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:17.160458 18112 tablet_bootstrap.cc:492] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: Bootstrap starting.
I20260924 07:16:17.160985 18111 tablet_bootstrap.cc:492] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: Bootstrap starting.
I20260924 07:16:17.161059 18110 tablet_bootstrap.cc:492] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: Bootstrap starting.
I20260924 07:16:17.161782 18112 tablet_bootstrap.cc:654] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:17.161957 18110 tablet_bootstrap.cc:654] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:17.161996 18111 tablet_bootstrap.cc:654] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:17.163187 18112 tablet_bootstrap.cc:492] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: No bootstrap required, opened a new log
I20260924 07:16:17.163215 18110 tablet_bootstrap.cc:492] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: No bootstrap required, opened a new log
I20260924 07:16:17.163295 18112 ts_tablet_manager.cc:1397] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:17.163307 18110 ts_tablet_manager.cc:1397] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:16:17.163704 18112 raft_consensus.cc:348] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.163710 18110 raft_consensus.cc:348] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.163818 18110 raft_consensus.cc:374] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:17.163822 18112 raft_consensus.cc:374] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:17.163862 18110 raft_consensus.cc:729] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105d2904d94c47718377372b1d68a9c6, State: Initialized, Role: FOLLOWER
I20260924 07:16:17.163869 18112 raft_consensus.cc:729] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Initialized, Role: FOLLOWER
I20260924 07:16:17.163986 18110 consensus_queue.cc:260] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.164000 18112 consensus_queue.cc:260] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.164283 18112 ts_tablet_manager.cc:1428] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:16:17.164283 18110 ts_tablet_manager.cc:1428] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260924 07:16:17.164250 18117 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.164382 18117 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.164498 18118 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.164619 18118 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.166065 18111 tablet_bootstrap.cc:492] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: No bootstrap required, opened a new log
I20260924 07:16:17.166170 18111 ts_tablet_manager.cc:1397] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 07:16:17.166687 18111 raft_consensus.cc:348] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.166805 18111 raft_consensus.cc:374] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:17.166837 18111 raft_consensus.cc:729] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de520ec398174145aca35db4306d1ff6, State: Initialized, Role: FOLLOWER
I20260924 07:16:17.166980 18111 consensus_queue.cc:260] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.167277 18116 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.167415 18116 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.167668 18111 ts_tablet_manager.cc:1428] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 07:16:17.387723 18116 raft_consensus.cc:482] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:17.387874 18116 raft_consensus.cc:504] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.389108 18116 leader_election.cc:288] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:17.393085 18109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.389507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:39547 (local address 127.15.127.129:32841)
0924 07:16:17.389768 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.389773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.389940 (+ 167us) server_negotiation.cc:407] Connection header received
0924 07:16:17.390107 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.390110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.390155 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.390241 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:17.390501 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.390988 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.392033 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.392418 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.392552 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.392603 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.392835 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.392908 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260924 07:16:17.393113 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.389399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:32841 (local address 127.15.127.131:39547)
0924 07:16:17.389829 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.389847 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.389980 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.390250 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.390253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.390268 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:17.390373 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.390384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.391097 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.391102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.391902 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.391910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.392547 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.392551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.392707 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.392724 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.392815 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.392895 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":133,"thread_start_us":65,"threads_started":1}
I20260924 07:16:17.393129 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.389530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:44077 (local address 127.15.127.130:42735)
0924 07:16:17.389798 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.389813 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.389828 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:17.389960 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.389963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.390003 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.390098 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:17.390381 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.390913 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.391976 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.392371 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.392512 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.392566 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.392887 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.392960 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260924 07:16:17.393374 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.389399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42735 (local address 127.15.127.131:44077)
0924 07:16:17.389697 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.389713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.389833 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.390109 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.390112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.390155 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:17.390269 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.390275 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.391022 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.391028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.391843 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.391859 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.392507 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.392510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.392610 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.392653 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.392873 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.392956 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260924 07:16:17.393556 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbdbecdf364ad09664a791969f9c09" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6" is_pre_election: true
I20260924 07:16:17.393743 17761 raft_consensus.cc:2454] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 0.
I20260924 07:16:17.393972 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbdbecdf364ad09664a791969f9c09" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4b1d96da3d4301bada9736ea36e421" is_pre_election: true
I20260924 07:16:17.394124 17974 leader_election.cc:302] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d2904d94c47718377372b1d68a9c6, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:17.394200 17901 raft_consensus.cc:2454] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 0.
I20260924 07:16:17.394450 18116 raft_consensus.cc:2781] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:17.394539 18116 raft_consensus.cc:482] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:17.394604 18116 raft_consensus.cc:3037] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.395599 18116 raft_consensus.cc:504] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.395923 18116 leader_election.cc:288] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:17.396262 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbdbecdf364ad09664a791969f9c09" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4b1d96da3d4301bada9736ea36e421"
I20260924 07:16:17.396250 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92fbdbecdf364ad09664a791969f9c09" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6"
I20260924 07:16:17.396363 17901 raft_consensus.cc:3037] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.396385 17761 raft_consensus.cc:3037] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.397454 17761 raft_consensus.cc:2454] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 1.
I20260924 07:16:17.397454 17901 raft_consensus.cc:2454] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 1.
I20260924 07:16:17.397729 17974 leader_election.cc:302] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d2904d94c47718377372b1d68a9c6, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:17.397909 18116 raft_consensus.cc:2781] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:17.398123 18116 raft_consensus.cc:686] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [term 1 LEADER]: Becoming Leader. State: Replica: de520ec398174145aca35db4306d1ff6, State: Running, Role: LEADER
I20260924 07:16:17.398480 18116 consensus_queue.cc:237] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.400710 18135 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: TxnStatusTablet state changed. Reason: New leader de520ec398174145aca35db4306d1ff6. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.400862 18135 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:17.401119 18137 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:16:17.401356 18137 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:16:17.402145 17603 catalog_manager.cc:5495] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de520ec398174145aca35db4306d1ff6 (127.15.127.131). New cstate: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } health_report { overall_health: HEALTHY } } }
I20260924 07:16:17.464570 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.460792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38845 (local address 127.0.0.1:39324)
0924 07:16:17.460940 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.460958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.461122 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.461479 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.461482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.461489 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:17.461603 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.461608 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.462333 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.462339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.462948 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.462956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.463418 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.463422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.463518 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.463565 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.464276 (+ 711us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.464291 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.464354 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.464440 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:16:17.464570 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.460951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39324 (local address 127.15.127.131:38845)
0924 07:16:17.461098 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.461103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.461118 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:17.461298 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.461301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.461366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.461468 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:17.461713 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.462225 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.463070 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.463309 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.463418 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.463735 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.464169 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.464291 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.464375 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.464428 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:16:17.470019 17761 raft_consensus.cc:1261] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Refusing update from remote peer de520ec398174145aca35db4306d1ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:17.470017 17901 raft_consensus.cc:1261] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Refusing update from remote peer de520ec398174145aca35db4306d1ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:17.470678 18135 consensus_queue.cc:1035] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:17.470916 18116 consensus_queue.cc:1035] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:17.474198 18118 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: TxnStatusTablet state changed. Reason: New leader de520ec398174145aca35db4306d1ff6. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.474319 18118 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.474820 18117 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: TxnStatusTablet state changed. Reason: New leader de520ec398174145aca35db4306d1ff6. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.474934 18117 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.477555 18135 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.477689 18135 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:17.477980 18118 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.478082 18118 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P 5e4b1d96da3d4301bada9736ea36e421: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.480124 18116 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.480243 18116 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P de520ec398174145aca35db4306d1ff6: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:17.481455 18142 mvcc.cc:205] Tried to move back new op lower bound from 7332799190909669376 to 7332799190626934784. Current Snapshot: MvccSnapshot[applied={T|T < 7332799190909669376}]
I20260924 07:16:17.481777 18141 mvcc.cc:205] Tried to move back new op lower bound from 7332799190909669376 to 7332799190626934784. Current Snapshot: MvccSnapshot[applied={T|T < 7332799190909669376}]
I20260924 07:16:17.481958 18117 tablet_replica.cc:439] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } } }
I20260924 07:16:17.482061 18117 tablet_replica.cc:442] T 92fbdbecdf364ad09664a791969f9c09 P 105d2904d94c47718377372b1d68a9c6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:17.491871 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.484045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40010 (local address 127.15.127.130:42735)
0924 07:16:17.484160 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.484165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.487756 (+ 3591us) server_negotiation.cc:407] Connection header received
0924 07:16:17.487904 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.487906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.487961 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.488088 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:17.488321 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.488823 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.490084 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.490352 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.490454 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.490863 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.491410 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.491533 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.491638 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.491709 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:16:17.504340 18152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.500653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38845 (local address 127.15.127.130:33601)
0924 07:16:17.501124 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.501137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.501271 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.501544 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.501547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.501555 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:17.501700 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.501706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.502440 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.502447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.503099 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.503108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.503548 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.503551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.503656 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.503698 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.504030 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.504039 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.504112 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.504186 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":125,"threads_started":1}
I20260924 07:16:17.504317 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.500753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:33601 (local address 127.15.127.131:38845)
0924 07:16:17.500880 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.500884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.501233 (+ 349us) server_negotiation.cc:407] Connection header received
0924 07:16:17.501386 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.501389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.501449 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.501533 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:17.501833 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.502334 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.503221 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.503438 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.503544 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.503879 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.503950 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.504027 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.504123 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.504174 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:17.512200 17761 raft_consensus.cc:1261] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Refusing update from remote peer 5e4b1d96da3d4301bada9736ea36e421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:17.512415 18036 raft_consensus.cc:1261] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Refusing update from remote peer 5e4b1d96da3d4301bada9736ea36e421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:17.513059 18118 consensus_queue.cc:1035] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:17.513262 18121 consensus_queue.cc:1035] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:17.520309 18140 mvcc.cc:205] Tried to move back new op lower bound from 7332799191085518848 to 7332799189394939904. Current Snapshot: MvccSnapshot[applied={T|T < 7332799191085518848}]
I20260924 07:16:17.538265 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.534489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42735 (local address 127.15.127.131:57289)
0924 07:16:17.534877 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.534912 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.535023 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.535308 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.535310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.535317 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:17.535422 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.535426 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.536308 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.536311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.536953 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.536961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.537495 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.537499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.537599 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.537641 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.537936 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.537945 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.538023 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.538110 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260924 07:16:17.538236 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.534606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:57289 (local address 127.15.127.130:42735)
0924 07:16:17.534759 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.534764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.534984 (+ 220us) server_negotiation.cc:407] Connection header received
0924 07:16:17.535136 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.535140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.535196 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.535288 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:17.535534 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.536180 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.537161 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.537384 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.537495 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.537776 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.537851 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.537931 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.538043 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.538092 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:16:17.567889 17602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:17.568125 17602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:17.571539 17602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Sending DeleteTablet for 3 replicas of tablet 289f31e9afc74413bd3d2c9dbb98a355
I20260924 07:16:17.572574 17730 tablet_service.cc:1514] Processing DeleteTablet for tablet 289f31e9afc74413bd3d2c9dbb98a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:17 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42972
I20260924 07:16:17.573226 17602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:17.573709 17602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:17.573788 17881 tablet_service.cc:1514] Processing DeleteTablet for tablet 289f31e9afc74413bd3d2c9dbb98a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:17 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40008
I20260924 07:16:17.574476 18021 tablet_service.cc:1514] Processing DeleteTablet for tablet 289f31e9afc74413bd3d2c9dbb98a355 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:17 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:39312
I20260924 07:16:17.574975 18160 tablet_replica.cc:330] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: stopping tablet replica
I20260924 07:16:17.575336 18160 raft_consensus.cc:2229] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:17.575618 18160 raft_consensus.cc:2258] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:17.575875 18161 tablet_replica.cc:330] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: stopping tablet replica
I20260924 07:16:17.577244 18162 tablet_replica.cc:330] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: stopping tablet replica
I20260924 07:16:17.577620 18161 raft_consensus.cc:2229] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:17.577930 18162 raft_consensus.cc:2229] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:17.579532 18160 ts_tablet_manager.cc:1905] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:17.579882 18021 tablet_service.cc:1467] Processing CreateTablet for tablet 88757bc499de45e6a09aeacdcb0e10a5 (DEFAULT_TABLE table=test-table [id=f1405e4c08dc47a19504ec915c034631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:17.580094 18162 raft_consensus.cc:2258] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:17.580339 18161 raft_consensus.cc:2258] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:17.580186 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88757bc499de45e6a09aeacdcb0e10a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:17.583083 18111 tablet_bootstrap.cc:492] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Bootstrap starting.
I20260924 07:16:17.580438 17881 tablet_service.cc:1467] Processing CreateTablet for tablet 88757bc499de45e6a09aeacdcb0e10a5 (DEFAULT_TABLE table=test-table [id=f1405e4c08dc47a19504ec915c034631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:17.583650 17881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88757bc499de45e6a09aeacdcb0e10a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:17.584081 18111 tablet_bootstrap.cc:654] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:17.585461 18111 tablet_bootstrap.cc:492] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: No bootstrap required, opened a new log
I20260924 07:16:17.585563 18111 ts_tablet_manager.cc:1397] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 07:16:17.585984 18111 raft_consensus.cc:348] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.586055 18112 tablet_bootstrap.cc:492] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Bootstrap starting.
I20260924 07:16:17.586103 18111 raft_consensus.cc:374] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:17.586138 18111 raft_consensus.cc:729] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de520ec398174145aca35db4306d1ff6, State: Initialized, Role: FOLLOWER
I20260924 07:16:17.586257 18111 consensus_queue.cc:260] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.586534 18111 ts_tablet_manager.cc:1428] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:17.587088 18112 tablet_bootstrap.cc:654] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:17.587411 18162 ts_tablet_manager.cc:1905] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:17.579573 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 88757bc499de45e6a09aeacdcb0e10a5 (DEFAULT_TABLE table=test-table [id=f1405e4c08dc47a19504ec915c034631]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:17.589604 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88757bc499de45e6a09aeacdcb0e10a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:17.590627 18161 ts_tablet_manager.cc:1905] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:17.592083 18110 tablet_bootstrap.cc:492] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Bootstrap starting.
I20260924 07:16:17.593166 18110 tablet_bootstrap.cc:654] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:17.594211 18160 ts_tablet_manager.cc:1918] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:17.594304 18160 log.cc:1196] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/289f31e9afc74413bd3d2c9dbb98a355
I20260924 07:16:17.594527 18110 tablet_bootstrap.cc:492] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: No bootstrap required, opened a new log
I20260924 07:16:17.594624 18160 ts_tablet_manager.cc:1939] T 289f31e9afc74413bd3d2c9dbb98a355 P 105d2904d94c47718377372b1d68a9c6: Deleting consensus metadata
I20260924 07:16:17.594640 18110 ts_tablet_manager.cc:1397] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:17.595208 18110 raft_consensus.cc:348] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.595328 18110 raft_consensus.cc:374] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:17.595361 18110 raft_consensus.cc:729] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105d2904d94c47718377372b1d68a9c6, State: Initialized, Role: FOLLOWER
I20260924 07:16:17.595739 18110 consensus_queue.cc:260] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.596128 17590 catalog_manager.cc:4841] TS 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841): tablet 289f31e9afc74413bd3d2c9dbb98a355 (table test-table [id=5a3bbd2eef9546628cbf9ac9e0c313f7]) successfully deleted
I20260924 07:16:17.596849 18110 ts_tablet_manager.cc:1428] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:17.598537 18162 ts_tablet_manager.cc:1918] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:17.598672 18162 log.cc:1196] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/289f31e9afc74413bd3d2c9dbb98a355
I20260924 07:16:17.599027 18162 ts_tablet_manager.cc:1939] T 289f31e9afc74413bd3d2c9dbb98a355 P de520ec398174145aca35db4306d1ff6: Deleting consensus metadata
I20260924 07:16:17.600160 17590 catalog_manager.cc:4841] TS de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845): tablet 289f31e9afc74413bd3d2c9dbb98a355 (table test-table [id=5a3bbd2eef9546628cbf9ac9e0c313f7]) successfully deleted
I20260924 07:16:17.601289 18161 ts_tablet_manager.cc:1918] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:17.601406 18161 log.cc:1196] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/289f31e9afc74413bd3d2c9dbb98a355
I20260924 07:16:17.601703 18161 ts_tablet_manager.cc:1939] T 289f31e9afc74413bd3d2c9dbb98a355 P 5e4b1d96da3d4301bada9736ea36e421: Deleting consensus metadata
I20260924 07:16:17.601876 18112 tablet_bootstrap.cc:492] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: No bootstrap required, opened a new log
I20260924 07:16:17.601976 18112 ts_tablet_manager.cc:1397] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260924 07:16:17.602445 18112 raft_consensus.cc:348] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.602582 18112 raft_consensus.cc:374] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:17.602615 18112 raft_consensus.cc:729] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Initialized, Role: FOLLOWER
I20260924 07:16:17.602735 18112 consensus_queue.cc:260] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.603001 18112 ts_tablet_manager.cc:1428] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:17.603251 17589 catalog_manager.cc:4841] TS 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735): tablet 289f31e9afc74413bd3d2c9dbb98a355 (table test-table [id=5a3bbd2eef9546628cbf9ac9e0c313f7]) successfully deleted
I20260924 07:16:17.786757 18153 raft_consensus.cc:482] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:17.786928 18153 raft_consensus.cc:504] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.787407 18153 leader_election.cc:288] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:17.787855 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88757bc499de45e6a09aeacdcb0e10a5" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6" is_pre_election: true
I20260924 07:16:17.787863 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88757bc499de45e6a09aeacdcb0e10a5" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4b1d96da3d4301bada9736ea36e421" is_pre_election: true
I20260924 07:16:17.788020 17761 raft_consensus.cc:2454] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 0.
I20260924 07:16:17.788024 17901 raft_consensus.cc:2454] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 0.
I20260924 07:16:17.788343 17973 leader_election.cc:302] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4b1d96da3d4301bada9736ea36e421, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:17.788537 18153 raft_consensus.cc:2781] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:17.788635 18153 raft_consensus.cc:482] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:17.788678 18153 raft_consensus.cc:3037] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.789798 18153 raft_consensus.cc:504] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.790230 18153 leader_election.cc:288] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:17.790613 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88757bc499de45e6a09aeacdcb0e10a5" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4b1d96da3d4301bada9736ea36e421"
I20260924 07:16:17.790613 17761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88757bc499de45e6a09aeacdcb0e10a5" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6"
I20260924 07:16:17.790750 17761 raft_consensus.cc:3037] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.790750 17901 raft_consensus.cc:3037] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:17.791922 17901 raft_consensus.cc:2454] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 1.
I20260924 07:16:17.791922 17761 raft_consensus.cc:2454] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 1.
I20260924 07:16:17.792265 17974 leader_election.cc:302] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d2904d94c47718377372b1d68a9c6, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:17.792448 18153 raft_consensus.cc:2781] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:17.792559 18153 raft_consensus.cc:686] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 1 LEADER]: Becoming Leader. State: Replica: de520ec398174145aca35db4306d1ff6, State: Running, Role: LEADER
I20260924 07:16:17.792716 18153 consensus_queue.cc:237] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:17.794539 17603 catalog_manager.cc:5495] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de520ec398174145aca35db4306d1ff6 (127.15.127.131). New cstate: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } health_report { overall_health: HEALTHY } } }
I20260924 07:16:17.834239 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.831313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39326 (local address 127.15.127.131:38845)
0924 07:16:17.831428 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.831439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.831451 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:17.831577 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.831579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.831619 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.831685 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:17.831891 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.832375 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.833186 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.833387 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.833461 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.833697 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.833758 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.833809 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.834088 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.834133 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:16:17.842998 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.839862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:41745 (local address 127.15.127.131:38845)
0924 07:16:17.839972 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:17.839975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:17.840001 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:16:17.840159 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:17.840161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:17.840199 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:17.840259 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:17.840444 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.841089 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.841890 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.842157 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.842236 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.842443 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.842502 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.842555 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:17.842827 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 07:16:17.842875 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:16:17.843022 18157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:17.839812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:38845 (local address 127.15.127.131:41745)
0924 07:16:17.839951 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:17.839964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:17.840070 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:17.840283 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:17.840285 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:17.840291 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:17.840364 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.840368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.841217 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.841220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.841794 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:17.841801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:17.842258 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:17.842261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:17.842321 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:17.842346 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:17.842578 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:17.842583 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:16:17.842795 (+ 212us) client_negotiation.cc:240] Negotiation successful
0924 07:16:17.842873 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:16:17.848711 17901 raft_consensus.cc:1261] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Refusing update from remote peer de520ec398174145aca35db4306d1ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:17.848698 17761 raft_consensus.cc:1261] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Refusing update from remote peer de520ec398174145aca35db4306d1ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:17.849123 18135 consensus_queue.cc:1035] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:17.849272 18116 consensus_queue.cc:1035] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:16:17.863754 18140 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:16:19.419245 17602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:19.419405 17602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:19.422344 17602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Sending DeleteTablet for 3 replicas of tablet 88757bc499de45e6a09aeacdcb0e10a5
I20260924 07:16:19.423036 18021 tablet_service.cc:1514] Processing DeleteTablet for tablet 88757bc499de45e6a09aeacdcb0e10a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:19 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:39312
I20260924 07:16:19.423058 17730 tablet_service.cc:1514] Processing DeleteTablet for tablet 88757bc499de45e6a09aeacdcb0e10a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:19 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42972
I20260924 07:16:19.423096 17881 tablet_service.cc:1514] Processing DeleteTablet for tablet 88757bc499de45e6a09aeacdcb0e10a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:19 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40008
I20260924 07:16:19.423415 18193 tablet_replica.cc:330] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: stopping tablet replica
I20260924 07:16:19.423451 18192 tablet_replica.cc:330] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: stopping tablet replica
I20260924 07:16:19.423591 18194 tablet_replica.cc:330] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: stopping tablet replica
I20260924 07:16:19.423632 18192 raft_consensus.cc:2229] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:19.423733 18194 raft_consensus.cc:2229] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:19.423893 18194 raft_consensus.cc:2258] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:19.424012 18192 raft_consensus.cc:2258] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:19.424137 18193 raft_consensus.cc:2229] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:19.424304 18193 raft_consensus.cc:2258] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:19.424852 18192 ts_tablet_manager.cc:1905] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:19.424866 18194 ts_tablet_manager.cc:1905] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:19.427333 18194 ts_tablet_manager.cc:1918] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:16:19.427433 18194 log.cc:1196] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/88757bc499de45e6a09aeacdcb0e10a5
I20260924 07:16:19.427894 18193 ts_tablet_manager.cc:1905] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:19.428471 18192 ts_tablet_manager.cc:1918] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:16:19.428555 18192 log.cc:1196] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/88757bc499de45e6a09aeacdcb0e10a5
I20260924 07:16:19.428843 18192 ts_tablet_manager.cc:1939] T 88757bc499de45e6a09aeacdcb0e10a5 P de520ec398174145aca35db4306d1ff6: Deleting consensus metadata
I20260924 07:16:19.430147 17590 catalog_manager.cc:4841] TS de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845): tablet 88757bc499de45e6a09aeacdcb0e10a5 (table test-table [id=f1405e4c08dc47a19504ec915c034631]) successfully deleted
I20260924 07:16:19.430727 18193 ts_tablet_manager.cc:1918] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:16:19.430835 18193 log.cc:1196] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/88757bc499de45e6a09aeacdcb0e10a5
I20260924 07:16:19.431133 18193 ts_tablet_manager.cc:1939] T 88757bc499de45e6a09aeacdcb0e10a5 P 105d2904d94c47718377372b1d68a9c6: Deleting consensus metadata
I20260924 07:16:19.431869 17590 catalog_manager.cc:4841] TS 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841): tablet 88757bc499de45e6a09aeacdcb0e10a5 (table test-table [id=f1405e4c08dc47a19504ec915c034631]) successfully deleted
I20260924 07:16:19.432343 18194 ts_tablet_manager.cc:1939] T 88757bc499de45e6a09aeacdcb0e10a5 P 5e4b1d96da3d4301bada9736ea36e421: Deleting consensus metadata
I20260924 07:16:19.433113 17589 catalog_manager.cc:4841] TS 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735): tablet 88757bc499de45e6a09aeacdcb0e10a5 (table test-table [id=f1405e4c08dc47a19504ec915c034631]) successfully deleted
Sep 24 07:16:19 dist-test-slave-zx46 krb5kdc[17559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:16:19 dist-test-slave-zx46 krb5kdc[17559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234179, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:19.454197 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:19.445355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.15.127.190:34489)
0924 07:16:19.445659 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:19.445666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:19.445737 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:16:19.445862 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:19.445865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:19.445908 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:19.445995 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:19.446697 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:19.447242 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:19.448058 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:19.448293 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:19.448376 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:19.451043 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:19.451046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:19.451048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:19.451099 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:19.452553 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:19.452962 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:19.452964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:19.452966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:19.453119 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:19.453359 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:19.453361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:19.453363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:19.453540 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:19.453674 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:19.453934 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 07:16:19.454041 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":125,"threads_started":1}
I20260924 07:16:19.456946 17602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:19.457453 17602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:19.462112 17881 tablet_service.cc:1467] Processing CreateTablet for tablet ab0210e8da9445f8a41e9d5fa232a6a0 (DEFAULT_TABLE table=test-table [id=ddd43045e5be43acb8755d74f0295441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:19.462119 17730 tablet_service.cc:1467] Processing CreateTablet for tablet ab0210e8da9445f8a41e9d5fa232a6a0 (DEFAULT_TABLE table=test-table [id=ddd43045e5be43acb8755d74f0295441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:19.462430 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0210e8da9445f8a41e9d5fa232a6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:19.462430 17881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0210e8da9445f8a41e9d5fa232a6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:19.464486 18203 tablet_bootstrap.cc:492] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Bootstrap starting.
I20260924 07:16:19.464862 18204 tablet_bootstrap.cc:492] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Bootstrap starting.
I20260924 07:16:19.464936 18021 tablet_service.cc:1467] Processing CreateTablet for tablet ab0210e8da9445f8a41e9d5fa232a6a0 (DEFAULT_TABLE table=test-table [id=ddd43045e5be43acb8755d74f0295441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:19.465219 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab0210e8da9445f8a41e9d5fa232a6a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:19.465837 18204 tablet_bootstrap.cc:654] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:19.465862 18203 tablet_bootstrap.cc:654] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:19.466984 18203 tablet_bootstrap.cc:492] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: No bootstrap required, opened a new log
I20260924 07:16:19.467070 18203 ts_tablet_manager.cc:1397] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:16:19.467567 18203 raft_consensus.cc:348] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.467686 18203 raft_consensus.cc:374] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:19.467708 18207 tablet_bootstrap.cc:492] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Bootstrap starting.
I20260924 07:16:19.467730 18203 raft_consensus.cc:729] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105d2904d94c47718377372b1d68a9c6, State: Initialized, Role: FOLLOWER
I20260924 07:16:19.467846 18203 consensus_queue.cc:260] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.468243 18203 ts_tablet_manager.cc:1428] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:19.468513 18207 tablet_bootstrap.cc:654] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:19.468811 18204 tablet_bootstrap.cc:492] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: No bootstrap required, opened a new log
I20260924 07:16:19.468905 18204 ts_tablet_manager.cc:1397] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 07:16:19.469465 18204 raft_consensus.cc:348] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.469589 18204 raft_consensus.cc:374] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:19.469646 18204 raft_consensus.cc:729] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Initialized, Role: FOLLOWER
I20260924 07:16:19.469805 18207 tablet_bootstrap.cc:492] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: No bootstrap required, opened a new log
I20260924 07:16:19.469781 18204 consensus_queue.cc:260] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.469964 18207 ts_tablet_manager.cc:1397] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 07:16:19.470182 18204 ts_tablet_manager.cc:1428] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 07:16:19.470453 18207 raft_consensus.cc:348] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.470626 18207 raft_consensus.cc:374] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:19.470683 18207 raft_consensus.cc:729] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de520ec398174145aca35db4306d1ff6, State: Initialized, Role: FOLLOWER
I20260924 07:16:19.470824 18207 consensus_queue.cc:260] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.471171 18207 ts_tablet_manager.cc:1428] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:19.772881 18186 raft_consensus.cc:482] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:19.773042 18186 raft_consensus.cc:504] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.773434 18186 leader_election.cc:288] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:19.773957 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0210e8da9445f8a41e9d5fa232a6a0" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4b1d96da3d4301bada9736ea36e421" is_pre_election: true
I20260924 07:16:19.774181 17901 raft_consensus.cc:2454] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 0.
I20260924 07:16:19.774173 17760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0210e8da9445f8a41e9d5fa232a6a0" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6" is_pre_election: true
I20260924 07:16:19.774348 17760 raft_consensus.cc:2454] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 0.
I20260924 07:16:19.774569 17973 leader_election.cc:302] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4b1d96da3d4301bada9736ea36e421, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:19.774807 18186 raft_consensus.cc:2781] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:19.774899 18186 raft_consensus.cc:482] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:19.774955 18186 raft_consensus.cc:3037] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:19.776072 18186 raft_consensus.cc:504] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.776453 18186 leader_election.cc:288] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 election: Requested vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735)
I20260924 07:16:19.776857 17760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0210e8da9445f8a41e9d5fa232a6a0" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6"
I20260924 07:16:19.777002 17760 raft_consensus.cc:3037] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:19.777132 17901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab0210e8da9445f8a41e9d5fa232a6a0" candidate_uuid: "de520ec398174145aca35db4306d1ff6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e4b1d96da3d4301bada9736ea36e421"
I20260924 07:16:19.777268 17901 raft_consensus.cc:3037] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:19.778292 17760 raft_consensus.cc:2454] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 1.
I20260924 07:16:19.778396 17901 raft_consensus.cc:2454] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de520ec398174145aca35db4306d1ff6 in term 1.
I20260924 07:16:19.778637 17974 leader_election.cc:302] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d2904d94c47718377372b1d68a9c6, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:19.778858 18186 raft_consensus.cc:2781] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:19.778954 18186 raft_consensus.cc:686] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 1 LEADER]: Becoming Leader. State: Replica: de520ec398174145aca35db4306d1ff6, State: Running, Role: LEADER
I20260924 07:16:19.779099 18186 consensus_queue.cc:237] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.780974 17604 catalog_manager.cc:5495] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 reported cstate change: term changed from 0 to 1, leader changed from <none> to de520ec398174145aca35db4306d1ff6 (127.15.127.131). New cstate: current_term: 1 leader_uuid: "de520ec398174145aca35db4306d1ff6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } health_report { overall_health: HEALTHY } } }
I20260924 07:16:19.839043 18213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:19.836074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.15.127.131:38845)
0924 07:16:19.836366 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:19.836372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:19.836384 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:19.836420 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:19.836422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:19.836463 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:19.836532 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:19.836761 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:19.837307 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:19.838121 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:19.838333 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:19.838404 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:19.838616 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:19.838678 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:19.838730 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:19.838809 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:16:19.838850 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":141,"threads_started":1}
I20260924 07:16:19.845808 17901 raft_consensus.cc:1261] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Refusing update from remote peer de520ec398174145aca35db4306d1ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:19.845808 17760 raft_consensus.cc:1261] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Refusing update from remote peer de520ec398174145aca35db4306d1ff6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:19.846159 18187 consensus_queue.cc:1035] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:19.846297 18153 consensus_queue.cc:1035] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:19.873520 17603 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:19.873677 17603 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:19.875485 17603 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Sending DeleteTablet for 3 replicas of tablet ab0210e8da9445f8a41e9d5fa232a6a0
I20260924 07:16:19.876008 17881 tablet_service.cc:1514] Processing DeleteTablet for tablet ab0210e8da9445f8a41e9d5fa232a6a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:19 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40008
I20260924 07:16:19.876144 17730 tablet_service.cc:1514] Processing DeleteTablet for tablet ab0210e8da9445f8a41e9d5fa232a6a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:19 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42972
I20260924 07:16:19.876214 18194 tablet_replica.cc:330] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: stopping tablet replica
I20260924 07:16:19.876255 18021 tablet_service.cc:1514] Processing DeleteTablet for tablet ab0210e8da9445f8a41e9d5fa232a6a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:19 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:39312
I20260924 07:16:19.876329 18193 tablet_replica.cc:330] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: stopping tablet replica
I20260924 07:16:19.876343 18194 raft_consensus.cc:2229] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:19.876433 18193 raft_consensus.cc:2229] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:19.876472 18194 raft_consensus.cc:2258] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:19.876549 18192 tablet_replica.cc:330] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: stopping tablet replica
I20260924 07:16:19.876540 18193 raft_consensus.cc:2258] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:19.876657 18192 raft_consensus.cc:2229] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:19.876837 18192 raft_consensus.cc:2258] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:19.877041 17603 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:16:19.877254 18193 ts_tablet_manager.cc:1905] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:19.877256 18194 ts_tablet_manager.cc:1905] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:16:19.877444 17603 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:19.877621 18192 ts_tablet_manager.cc:1905] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:19.879833 18194 ts_tablet_manager.cc:1918] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:19.879838 18193 ts_tablet_manager.cc:1918] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:19.879914 18193 log.cc:1196] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/ab0210e8da9445f8a41e9d5fa232a6a0
I20260924 07:16:19.879915 18194 log.cc:1196] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/ab0210e8da9445f8a41e9d5fa232a6a0
I20260924 07:16:19.880190 18194 ts_tablet_manager.cc:1939] T ab0210e8da9445f8a41e9d5fa232a6a0 P 5e4b1d96da3d4301bada9736ea36e421: Deleting consensus metadata
I20260924 07:16:19.880182 18192 ts_tablet_manager.cc:1918] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:19.880249 18193 ts_tablet_manager.cc:1939] T ab0210e8da9445f8a41e9d5fa232a6a0 P 105d2904d94c47718377372b1d68a9c6: Deleting consensus metadata
I20260924 07:16:19.880247 18192 log.cc:1196] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/ab0210e8da9445f8a41e9d5fa232a6a0
I20260924 07:16:19.880508 18192 ts_tablet_manager.cc:1939] T ab0210e8da9445f8a41e9d5fa232a6a0 P de520ec398174145aca35db4306d1ff6: Deleting consensus metadata
I20260924 07:16:19.880920 17589 catalog_manager.cc:4841] TS 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130:42735): tablet ab0210e8da9445f8a41e9d5fa232a6a0 (table test-table [id=ddd43045e5be43acb8755d74f0295441]) successfully deleted
I20260924 07:16:19.880965 17590 catalog_manager.cc:4841] TS 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841): tablet ab0210e8da9445f8a41e9d5fa232a6a0 (table test-table [id=ddd43045e5be43acb8755d74f0295441]) successfully deleted
I20260924 07:16:19.881613 17590 catalog_manager.cc:4841] TS de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845): tablet ab0210e8da9445f8a41e9d5fa232a6a0 (table test-table [id=ddd43045e5be43acb8755d74f0295441]) successfully deleted
I20260924 07:16:19.882449 17730 tablet_service.cc:1467] Processing CreateTablet for tablet 8e930c2df77449869010ef85215d9aeb (DEFAULT_TABLE table=test-table [id=39707c7c0ba04cb38ae9c52f7d63b831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:19.882722 17730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e930c2df77449869010ef85215d9aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:19.882721 17881 tablet_service.cc:1467] Processing CreateTablet for tablet 8e930c2df77449869010ef85215d9aeb (DEFAULT_TABLE table=test-table [id=39707c7c0ba04cb38ae9c52f7d63b831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:19.882908 17881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e930c2df77449869010ef85215d9aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:19.884435 18203 tablet_bootstrap.cc:492] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6: Bootstrap starting.
I20260924 07:16:19.885178 18204 tablet_bootstrap.cc:492] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: Bootstrap starting.
I20260924 07:16:19.885347 18203 tablet_bootstrap.cc:654] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:19.886009 18204 tablet_bootstrap.cc:654] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:19.886224 18021 tablet_service.cc:1467] Processing CreateTablet for tablet 8e930c2df77449869010ef85215d9aeb (DEFAULT_TABLE table=test-table [id=39707c7c0ba04cb38ae9c52f7d63b831]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:19.886423 18203 tablet_bootstrap.cc:492] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6: No bootstrap required, opened a new log
I20260924 07:16:19.886499 18203 ts_tablet_manager.cc:1397] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:19.886534 18021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e930c2df77449869010ef85215d9aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:19.886977 18203 raft_consensus.cc:348] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.887122 18203 raft_consensus.cc:374] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:19.887162 18203 raft_consensus.cc:729] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 105d2904d94c47718377372b1d68a9c6, State: Initialized, Role: FOLLOWER
I20260924 07:16:19.887228 18204 tablet_bootstrap.cc:492] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: No bootstrap required, opened a new log
I20260924 07:16:19.887297 18204 ts_tablet_manager.cc:1397] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 07:16:19.887285 18203 consensus_queue.cc:260] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.887573 18203 ts_tablet_manager.cc:1428] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:19.887702 18204 raft_consensus.cc:348] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.887836 18204 raft_consensus.cc:374] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:19.887882 18204 raft_consensus.cc:729] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Initialized, Role: FOLLOWER
I20260924 07:16:19.888015 18204 consensus_queue.cc:260] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.888271 18204 ts_tablet_manager.cc:1428] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260924 07:16:19.888949 18207 tablet_bootstrap.cc:492] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Bootstrap starting.
I20260924 07:16:19.889750 18207 tablet_bootstrap.cc:654] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:19.890944 18207 tablet_bootstrap.cc:492] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: No bootstrap required, opened a new log
I20260924 07:16:19.891036 18207 ts_tablet_manager.cc:1397] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:19.891388 18207 raft_consensus.cc:348] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.891480 18207 raft_consensus.cc:374] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:19.891501 18207 raft_consensus.cc:729] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de520ec398174145aca35db4306d1ff6, State: Initialized, Role: FOLLOWER
I20260924 07:16:19.891650 18207 consensus_queue.cc:260] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:19.891893 18207 ts_tablet_manager.cc:1428] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:16:20.013453 18210 raft_consensus.cc:482] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:20.013628 18210 raft_consensus.cc:504] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:20.014061 18210 leader_election.cc:288] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845)
I20260924 07:16:20.014719 17760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e930c2df77449869010ef85215d9aeb" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6" is_pre_election: true
I20260924 07:16:20.014719 18036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e930c2df77449869010ef85215d9aeb" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de520ec398174145aca35db4306d1ff6" is_pre_election: true
I20260924 07:16:20.014880 17760 raft_consensus.cc:2454] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 0.
I20260924 07:16:20.014880 18036 raft_consensus.cc:2454] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 0.
I20260924 07:16:20.015204 17834 leader_election.cc:302] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e4b1d96da3d4301bada9736ea36e421, de520ec398174145aca35db4306d1ff6; no voters:
I20260924 07:16:20.015408 18210 raft_consensus.cc:2781] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:20.015496 18210 raft_consensus.cc:482] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:20.015552 18210 raft_consensus.cc:3037] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:20.016896 18210 raft_consensus.cc:504] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:20.017304 18210 leader_election.cc:288] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 election: Requested vote from peers 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841), de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845)
I20260924 07:16:20.017827 17760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e930c2df77449869010ef85215d9aeb" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "105d2904d94c47718377372b1d68a9c6"
I20260924 07:16:20.017968 17760 raft_consensus.cc:3037] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:20.018100 18036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e930c2df77449869010ef85215d9aeb" candidate_uuid: "5e4b1d96da3d4301bada9736ea36e421" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de520ec398174145aca35db4306d1ff6"
I20260924 07:16:20.018289 18036 raft_consensus.cc:3037] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:20.019140 17760 raft_consensus.cc:2454] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 1.
I20260924 07:16:20.019474 17834 leader_election.cc:302] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 105d2904d94c47718377372b1d68a9c6, 5e4b1d96da3d4301bada9736ea36e421; no voters:
I20260924 07:16:20.019635 18210 raft_consensus.cc:2781] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:20.019781 18210 raft_consensus.cc:686] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 1 LEADER]: Becoming Leader. State: Replica: 5e4b1d96da3d4301bada9736ea36e421, State: Running, Role: LEADER
I20260924 07:16:20.019936 18210 consensus_queue.cc:237] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } }
I20260924 07:16:20.020900 18036 raft_consensus.cc:2454] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e4b1d96da3d4301bada9736ea36e421 in term 1.
I20260924 07:16:20.022265 17604 catalog_manager.cc:5495] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e4b1d96da3d4301bada9736ea36e421 (127.15.127.130). New cstate: current_term: 1 leader_uuid: "5e4b1d96da3d4301bada9736ea36e421" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e4b1d96da3d4301bada9736ea36e421" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 } health_report { overall_health: UNKNOWN } } }
I20260924 07:16:20.052268 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:20.049196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.15.127.130:42735)
0924 07:16:20.049475 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:20.049482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:20.049494 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:20.049535 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:20.049537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:20.049578 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:20.049651 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:20.049836 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:20.050402 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:20.051231 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:20.051464 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:20.051543 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:20.051767 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:20.051862 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:20.051918 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:20.052080 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:16:20.052139 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
I20260924 07:16:20.059245 17760 raft_consensus.cc:1261] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6 [term 1 FOLLOWER]: Refusing update from remote peer 5e4b1d96da3d4301bada9736ea36e421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:20.059656 18036 raft_consensus.cc:1261] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Refusing update from remote peer 5e4b1d96da3d4301bada9736ea36e421: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:20.059736 18210 consensus_queue.cc:1035] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "105d2904d94c47718377372b1d68a9c6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 32841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:20.060075 18210 consensus_queue.cc:1035] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de520ec398174145aca35db4306d1ff6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 38845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:16:20.074340 18175 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:16:21.627135 17604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:21.627287 17604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:21.629693 17604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5610b6156c604c189791275d4dcec374: Sending DeleteTablet for 3 replicas of tablet 8e930c2df77449869010ef85215d9aeb
I20260924 07:16:21.630290 17730 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e930c2df77449869010ef85215d9aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:21 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42972
I20260924 07:16:21.630311 17881 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e930c2df77449869010ef85215d9aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:21 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40008
I20260924 07:16:21.630321 18021 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e930c2df77449869010ef85215d9aeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:21 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:39312
I20260924 07:16:21.630697 18250 tablet_replica.cc:330] T 8e930c2df77449869010ef85215d9aeb P 105d2904d94c47718377372b1d68a9c6: stopping tablet replica
I20260924 07:16:21.630733 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17676
I20260924 07:16:21.631587 18248 tablet_replica.cc:330] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: stopping tablet replica
I20260924 07:16:21.631738 18248 raft_consensus.cc:2229] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:21.631860 18248 raft_consensus.cc:2258] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:21.630636 18249 tablet_replica.cc:330] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: stopping tablet replica
I20260924 07:16:21.632095 18249 raft_consensus.cc:2229] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:21.632306 18249 raft_consensus.cc:2258] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:21.633086 18248 ts_tablet_manager.cc:1905] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:21.635221 18248 ts_tablet_manager.cc:1918] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:16:21.635320 18248 log.cc:1196] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/8e930c2df77449869010ef85215d9aeb
I20260924 07:16:21.635623 18248 ts_tablet_manager.cc:1939] T 8e930c2df77449869010ef85215d9aeb P de520ec398174145aca35db4306d1ff6: Deleting consensus metadata
I20260924 07:16:21.635851 18249 ts_tablet_manager.cc:1905] T 8e930c2df77449869010ef85215d9aeb P 5e4b1d96da3d4301bada9736ea36e421: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:21.636523 17590 catalog_manager.cc:4841] TS de520ec398174145aca35db4306d1ff6 (127.15.127.131:38845): tablet 8e930c2df77449869010ef85215d9aeb (table test-table [id=39707c7c0ba04cb38ae9c52f7d63b831]) successfully deleted
W20260924 07:16:21.637758 17590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:32841 (error 108)
I20260924 07:16:21.637815 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17816
I20260924 07:16:21.639861 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:21.639305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:32841 (local address 127.0.0.1:42982)
0924 07:16:21.639635 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:21.639725 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:32841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":82,"threads_started":1}
W20260924 07:16:21.640146 17590 catalog_manager.cc:4584] TS 105d2904d94c47718377372b1d68a9c6 (127.15.127.129:32841): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e930c2df77449869010ef85215d9aeb: Network error: Client connection negotiation failed: client connection to 127.15.127.129:32841: connect: Connection refused (error 111)
I20260924 07:16:21.645779 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:21.645527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42735 (local address 127.0.0.1:40024)
0924 07:16:21.645654 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:21.645703 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:42735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 07:16:21.646413 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17956
I20260924 07:16:21.653663 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 17572
2026-09-24T07:16:21Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7007 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:21 dist-test-slave-zx46 krb5kdc[18255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:21 dist-test-slave-zx46 krb5kdc[18255](info): set up 2 sockets
Sep 24 07:16:21 dist-test-slave-zx46 krb5kdc[18255](info): commencing operation
krb5kdc: starting...
W20260924 07:16:22.704910 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:22 dist-test-slave-zx46 krb5kdc[18255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:22.829407 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41329
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:22.942401 18268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:22.942768 18268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:22.942855 18268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:22.947047 18268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:22.947152 18268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:22.947196 18268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:22.947263 18268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:22.947307 18268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:22.952857 18268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:41329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:22.954151 18268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:22.955286 18268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:16:22.964919 18268 server_base.cc:1034] running on GCE node
W20260924 07:16:22.964844 18274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:22.964869 18276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:22.964847 18273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:22.965855 18268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:22.967139 18268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:22.968362 18268 hybrid_clock.cc:648] HybridClock initialized: now 1790234182968322 us; error 46 us; skew 500 ppm
Sep 24 07:16:22 dist-test-slave-zx46 krb5kdc[18255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234182, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:22.971603 18268 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:22.972527 18268 webserver.cc:460] Webserver started at http://127.15.127.190:41059/ using document root <none> and password file <none>
I20260924 07:16:22.973182 18268 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:22.973275 18268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:22.973515 18268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:22.975669 18268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "f361b9ef874649eab069e87d65afa452"
format_stamp: "Formatted at 2026-09-24 07:16:22 on dist-test-slave-zx46"
server_key: "27ccffdc3e077164c7048bba9793d7be"
server_key_iv: "d0dfb4f5910d735ecf9d4df1e48b967c"
server_key_version: "encryptionkey@0"
I20260924 07:16:22.976270 18268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "f361b9ef874649eab069e87d65afa452"
format_stamp: "Formatted at 2026-09-24 07:16:22 on dist-test-slave-zx46"
server_key: "27ccffdc3e077164c7048bba9793d7be"
server_key_iv: "d0dfb4f5910d735ecf9d4df1e48b967c"
server_key_version: "encryptionkey@0"
I20260924 07:16:22.980202 18268 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:16:22.982729 18283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:22.984112 18268 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:22.984265 18268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "f361b9ef874649eab069e87d65afa452"
format_stamp: "Formatted at 2026-09-24 07:16:22 on dist-test-slave-zx46"
server_key: "27ccffdc3e077164c7048bba9793d7be"
server_key_iv: "d0dfb4f5910d735ecf9d4df1e48b967c"
server_key_version: "encryptionkey@0"
I20260924 07:16:22.984385 18268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:23.002549 18268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:23.005638 18268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:23.005839 18268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:23.013200 18268 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:41329
I20260924 07:16:23.013253 18335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:41329 every 8 connection(s)
I20260924 07:16:23.014195 18268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:23.015866 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18268
I20260924 07:16:23.015959 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:23.016234 15870 external_mini_cluster.cc:1428] Setting key 0de6d5f6142d5b4eed2ea190bdb9fd94
I20260924 07:16:23.017325 18336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:23.024084 18336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452: Bootstrap starting.
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:23.027009 18336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:23.028014 18336 log.cc:824] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:23.030439 18336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452: No bootstrap required, opened a new log
I20260924 07:16:23.031989 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.017746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.15.127.190:41329)
0924 07:16:23.018168 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:23.018187 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:23.018230 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:16:23.019081 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:23.019104 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:23.019519 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:23.019896 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:23.020592 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.022082 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.022958 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.023641 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.023804 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.026631 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:23.026638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:23.026656 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:23.026691 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:23.028705 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.029255 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.029261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.029268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.029400 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.029684 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.029688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.029689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.030216 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:23.030402 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:23.030625 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 07:16:23.030970 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260924 07:16:23.034528 18336 raft_consensus.cc:348] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f361b9ef874649eab069e87d65afa452" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41329 } }
I20260924 07:16:23.034770 18336 raft_consensus.cc:374] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:23.034804 18336 raft_consensus.cc:729] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f361b9ef874649eab069e87d65afa452, State: Initialized, Role: FOLLOWER
I20260924 07:16:23.035166 18336 consensus_queue.cc:260] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f361b9ef874649eab069e87d65afa452" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41329 } }
I20260924 07:16:23.035272 18336 raft_consensus.cc:388] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:23.035312 18336 raft_consensus.cc:482] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:23.035382 18336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:23.036294 18336 raft_consensus.cc:504] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f361b9ef874649eab069e87d65afa452" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41329 } }
I20260924 07:16:23.036604 18336 leader_election.cc:302] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f361b9ef874649eab069e87d65afa452; no voters:
I20260924 07:16:23.036960 18336 leader_election.cc:288] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:23.037240 18341 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:23.037485 18341 raft_consensus.cc:686] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [term 1 LEADER]: Becoming Leader. State: Replica: f361b9ef874649eab069e87d65afa452, State: Running, Role: LEADER
I20260924 07:16:23.037842 18341 consensus_queue.cc:237] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f361b9ef874649eab069e87d65afa452" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41329 } }
I20260924 07:16:23.038347 18336 sys_catalog.cc:564] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:23.039009 18341 sys_catalog.cc:455] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f361b9ef874649eab069e87d65afa452. Latest consensus state: current_term: 1 leader_uuid: "f361b9ef874649eab069e87d65afa452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f361b9ef874649eab069e87d65afa452" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41329 } } }
I20260924 07:16:23.039225 18341 sys_catalog.cc:458] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:23.039428 18342 sys_catalog.cc:455] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f361b9ef874649eab069e87d65afa452" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f361b9ef874649eab069e87d65afa452" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41329 } } }
I20260924 07:16:23.039564 18342 sys_catalog.cc:458] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:23.039757 18350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:23.043167 18350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:23.049467 18350 catalog_manager.cc:1347] Generated new cluster ID: beba3ec0bd604ae983d4c4213310ff38
I20260924 07:16:23.049589 18350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:23.072886 18350 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:23.074170 18350 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:23.081053 18350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f361b9ef874649eab069e87d65afa452: Generated new TSK 0
I20260924 07:16:23.081820 18350 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:23.117169 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41329
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:23.234040 18362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:23.234336 18362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:23.234385 18362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:23.238656 18362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:23.238744 18362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:23.238811 18362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:23.244638 18362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:23.245910 18362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:23.247054 18362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:23.257321 18368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:23.257344 18367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:23.257404 18370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:23.257784 18362 server_base.cc:1034] running on GCE node
I20260924 07:16:23.258253 18362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:23.259009 18362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:23.260197 18362 hybrid_clock.cc:648] HybridClock initialized: now 1790234183260165 us; error 42 us; skew 500 ppm
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234183, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:23.263449 18362 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:23.264496 18362 webserver.cc:460] Webserver started at http://127.15.127.129:43955/ using document root <none> and password file <none>
I20260924 07:16:23.265195 18362 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:23.265273 18362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:23.265529 18362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:23.267735 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "6a1e4e40163148ff857259c66f6f4127"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "4cd3a5b01994e18c998d62774c7be251"
server_key_iv: "26b69610bbb29a626a2ff6e2b2612954"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.268359 18362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "6a1e4e40163148ff857259c66f6f4127"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "4cd3a5b01994e18c998d62774c7be251"
server_key_iv: "26b69610bbb29a626a2ff6e2b2612954"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.272573 18362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:16:23.275378 18377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.276721 18362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:23.276865 18362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "6a1e4e40163148ff857259c66f6f4127"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "4cd3a5b01994e18c998d62774c7be251"
server_key_iv: "26b69610bbb29a626a2ff6e2b2612954"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.277036 18362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:23.292870 18362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:23.296380 18362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:23.296550 18362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:23.297214 18362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:23.298024 18362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:23.298075 18362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.298116 18362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:23.298132 18362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.307487 18362 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:40651
I20260924 07:16:23.307505 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:40651 every 8 connection(s)
I20260924 07:16:23.308416 18362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:23.314107 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18362
I20260924 07:16:23.314235 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:23.314484 15870 external_mini_cluster.cc:1428] Setting key 66f98f9a33becba6b3a7485d6651c87b
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234183, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:23.323212 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.310289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:33483 (local address 127.15.127.190:41329)
0924 07:16:23.310445 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:23.310450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:23.311385 (+ 935us) server_negotiation.cc:407] Connection header received
0924 07:16:23.312630 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:23.312642 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:23.312691 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:23.312770 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:23.314016 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.314648 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.316109 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.316335 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.316414 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.319133 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:23.319140 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:23.319146 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:23.319181 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:23.320840 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.321525 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.321527 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.321529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.321608 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.322029 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.322031 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.322032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.322218 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:23.322292 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:23.322963 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 07:16:23.323087 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:23.324504 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.310753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41329 (local address 127.15.127.129:33483)
0924 07:16:23.311219 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:23.311266 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:23.312419 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:23.312938 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:23.312945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:23.313342 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:23.313855 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:23.313875 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.314777 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.314781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:23.315975 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:23.315986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.316489 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.316496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:23.316731 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.317388 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:23.317414 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:23.318957 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:23.320963 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:23.320974 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:23.320994 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:23.321419 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:23.321706 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:23.321713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:23.321716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:23.321927 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:23.322326 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:23.322333 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:23.322693 (+ 360us) client_negotiation.cc:769] Sending connection context
0924 07:16:23.322921 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 07:16:23.323298 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:23.325889 18491 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41329
I20260924 07:16:23.326118 18491 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:23.326598 18491 heartbeater.cc:511] Master 127.15.127.190:41329 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:16:23.328393 18300 ts_manager.cc:194] Registered new tserver with Master: 6a1e4e40163148ff857259c66f6f4127 (127.15.127.129:40651)
I20260924 07:16:23.329846 18300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:33483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:23.344035 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41329
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:23.455770 18496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:23.456066 18496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:23.456111 18496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:23.460348 18496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:23.460414 18496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:23.460525 18496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:23.466020 18496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:23.467203 18496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:23.468272 18496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:23.478657 18502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:23.478724 18501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:23.478672 18504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:23.478863 18496 server_base.cc:1034] running on GCE node
I20260924 07:16:23.479471 18496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:23.480187 18496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:23.481370 18496 hybrid_clock.cc:648] HybridClock initialized: now 1790234183481341 us; error 34 us; skew 500 ppm
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234183, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:23.484083 18496 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:23.484978 18496 webserver.cc:460] Webserver started at http://127.15.127.130:33535/ using document root <none> and password file <none>
I20260924 07:16:23.485620 18496 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:23.485714 18496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:23.485954 18496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:23.488140 18496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "5076bdce44264b14b3ee893739206e0c"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "bf768948401cc2775bbe473830446f33"
server_key_iv: "a7c175984c797b74302f425804e12cea"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.488771 18496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "5076bdce44264b14b3ee893739206e0c"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "bf768948401cc2775bbe473830446f33"
server_key_iv: "a7c175984c797b74302f425804e12cea"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.492652 18496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 07:16:23.495226 18511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.496493 18496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:23.496692 18496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "5076bdce44264b14b3ee893739206e0c"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "bf768948401cc2775bbe473830446f33"
server_key_iv: "a7c175984c797b74302f425804e12cea"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.496826 18496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:23.509423 18496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:23.512346 18496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:23.512563 18496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:23.513360 18496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:23.514472 18496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:23.514551 18496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.514622 18496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:23.514647 18496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.524089 18496 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:35821
I20260924 07:16:23.524194 18624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:35821 every 8 connection(s)
I20260924 07:16:23.525079 18496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:23.530601 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18496
I20260924 07:16:23.530728 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:23.531046 15870 external_mini_cluster.cc:1428] Setting key 955ca3626a36e85d71946d121a6e4519
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234183, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:23.540189 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.526871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:42353 (local address 127.15.127.190:41329)
0924 07:16:23.527019 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:23.527022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:23.527857 (+ 835us) server_negotiation.cc:407] Connection header received
0924 07:16:23.529143 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:23.529146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:23.529194 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:23.529270 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:23.530435 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.531042 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.532429 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.532738 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.532852 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.535552 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:23.535555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:23.535557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:23.535583 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:23.537189 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.537867 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.537869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.537871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.537945 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.538465 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.538467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.538468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.539010 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:23.539104 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:23.539928 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 07:16:23.540061 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:16:23.541489 18627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.527263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41329 (local address 127.15.127.130:42353)
0924 07:16:23.527698 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:23.527740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:23.528866 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:23.529431 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:23.529438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:23.529802 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:23.530280 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:23.530293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.531236 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.531239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:23.532289 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:23.532300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.532889 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.532892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:23.533118 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.533777 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:23.533795 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:23.535383 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:23.537302 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:23.537312 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:23.537333 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:23.537734 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:23.538084 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:23.538087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:23.538089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:23.538350 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:23.539135 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:23.539143 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:23.539543 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 07:16:23.539848 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 07:16:23.540264 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:23.542804 18625 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41329
I20260924 07:16:23.543084 18625 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:23.543619 18625 heartbeater.cc:511] Master 127.15.127.190:41329 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:23.544771 18300 ts_manager.cc:194] Registered new tserver with Master: 5076bdce44264b14b3ee893739206e0c (127.15.127.130:35821)
I20260924 07:16:23.545439 18300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:42353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:23.560575 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41329
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:23.673900 18630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:23.674237 18630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:23.674314 18630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:23.678421 18630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:23.678526 18630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:23.678622 18630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:23.684248 18630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:23.685441 18630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:23.686515 18630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:16:23.696946 18630 server_base.cc:1034] running on GCE node
W20260924 07:16:23.696979 18636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:23.696979 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:23.696938 18635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:23.697599 18630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:23.698210 18630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:23.699359 18630 hybrid_clock.cc:648] HybridClock initialized: now 1790234183699335 us; error 38 us; skew 500 ppm
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234183, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:23.702313 18630 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:23.703207 18630 webserver.cc:460] Webserver started at http://127.15.127.131:38191/ using document root <none> and password file <none>
I20260924 07:16:23.703789 18630 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:23.703851 18630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:23.704082 18630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:23.706198 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "1a6059a0cc5a4ab8b1ed44fa5664fe7e"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "1c3b7dad0232226acf6edade70f3e076"
server_key_iv: "26de17a58799108f823b7cf8207d2465"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.706789 18630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "1a6059a0cc5a4ab8b1ed44fa5664fe7e"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "1c3b7dad0232226acf6edade70f3e076"
server_key_iv: "26de17a58799108f823b7cf8207d2465"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.710481 18630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:16:23.712903 18645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.714193 18630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:23.714332 18630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "1a6059a0cc5a4ab8b1ed44fa5664fe7e"
format_stamp: "Formatted at 2026-09-24 07:16:23 on dist-test-slave-zx46"
server_key: "1c3b7dad0232226acf6edade70f3e076"
server_key_iv: "26de17a58799108f823b7cf8207d2465"
server_key_version: "encryptionkey@0"
I20260924 07:16:23.714444 18630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:23.730927 18630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:23.734468 18630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:23.734692 18630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:23.735301 18630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:23.736151 18630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:23.736223 18630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.736295 18630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:23.736348 18630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:23.745335 18630 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:42481
I20260924 07:16:23.745397 18758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:42481 every 8 connection(s)
I20260924 07:16:23.746296 18630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:23.747400 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18630
I20260924 07:16:23.747531 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:23.747879 15870 external_mini_cluster.cc:1428] Setting key 3611578728180840e544f0f45ad9ca5c
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234183, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:23.760289 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.748324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:53739 (local address 127.15.127.190:41329)
0924 07:16:23.748461 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:23.748472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:23.749317 (+ 845us) server_negotiation.cc:407] Connection header received
0924 07:16:23.750864 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:23.750867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:23.750924 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:23.750996 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:23.752263 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.752807 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.754028 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.754264 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.754376 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.756943 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:23.756946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:23.756948 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:23.756976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:23.758394 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.758914 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.758917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.758918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.758995 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:23.759298 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:23.759301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:23.759302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:23.759476 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:23.759550 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:23.760046 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 07:16:23.760165 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:16:23.761312 18761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:23.748705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41329 (local address 127.15.127.131:53739)
0924 07:16:23.749160 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:23.749202 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:23.750632 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:23.751195 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:23.751205 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:23.751629 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:23.752117 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:23.752130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.752937 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.752941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:23.753902 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:23.753911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:23.754414 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:23.754418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:23.754607 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:23.755219 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:23.755235 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:23.756755 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:23.758520 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:23.758525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:23.758540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:23.758819 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:23.759082 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:23.759084 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:23.759085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:23.759223 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:23.759566 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:23.759572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:23.759834 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 07:16:23.760002 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 07:16:23.760293 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260924 07:16:23.762887 18759 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41329
I20260924 07:16:23.763199 18759 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:23.763724 18759 heartbeater.cc:511] Master 127.15.127.190:41329 requested a full tablet report, sending...
I20260924 07:16:23.764773 18299 ts_manager.cc:194] Registered new tserver with Master: 1a6059a0cc5a4ab8b1ed44fa5664fe7e (127.15.127.131:42481)
I20260924 07:16:23.765462 18299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:53739
I20260924 07:16:23.775393 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:16:23.788552 18339 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:23.787046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.15.127.190:41329)
0924 07:16:23.787243 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:23.787248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:23.787319 (+ 71us) server_negotiation.cc:407] Connection header received
0924 07:16:23.787438 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:23.787441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:23.787487 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:23.787560 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:23.788395 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39020: BlockingRecv error: recv got EOF from 127.0.0.1:39020 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260924 07:16:23.789600 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18362
I20260924 07:16:23.795023 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18496
I20260924 07:16:23.800181 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18630
I20260924 07:16:23.805348 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18268
2026-09-24T07:16:23Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2149 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18771](info): set up 2 sockets
Sep 24 07:16:23 dist-test-slave-zx46 krb5kdc[18771](info): commencing operation
krb5kdc: starting...
W20260924 07:16:24.849184 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:24 dist-test-slave-zx46 krb5kdc[18771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:24.970317 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36191
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:45575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:36191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:25.083469 18784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:25.083801 18784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:25.083874 18784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:25.088049 18784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:25.088145 18784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:25.088187 18784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:25.088248 18784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:25.088289 18784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:25.093672 18784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:36191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:25.094764 18784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:25.095846 18784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:25.104285 18790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:25.104403 18792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:25.104336 18789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:25.104529 18784 server_base.cc:1034] running on GCE node
I20260924 07:16:25.105235 18784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:25.106389 18784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:25.107591 18784 hybrid_clock.cc:648] HybridClock initialized: now 1790234185107574 us; error 28 us; skew 500 ppm
Sep 24 07:16:25 dist-test-slave-zx46 krb5kdc[18771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234185, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:25.110484 18784 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:25.111339 18784 webserver.cc:460] Webserver started at http://127.15.127.190:38233/ using document root <none> and password file <none>
I20260924 07:16:25.111893 18784 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:25.111982 18784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:25.112210 18784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:25.114264 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "1ad5ab2e38034e8eb81966dd92513b5d"
format_stamp: "Formatted at 2026-09-24 07:16:25 on dist-test-slave-zx46"
server_key: "c885f5cce2e094311d487f69eedf5cad"
server_key_iv: "39e07a2dec9e8ec6ba4aa74d526d8318"
server_key_version: "encryptionkey@0"
I20260924 07:16:25.114828 18784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "1ad5ab2e38034e8eb81966dd92513b5d"
format_stamp: "Formatted at 2026-09-24 07:16:25 on dist-test-slave-zx46"
server_key: "c885f5cce2e094311d487f69eedf5cad"
server_key_iv: "39e07a2dec9e8ec6ba4aa74d526d8318"
server_key_version: "encryptionkey@0"
I20260924 07:16:25.118517 18784 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:25.120947 18799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:25.122188 18784 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:25.122328 18784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "1ad5ab2e38034e8eb81966dd92513b5d"
format_stamp: "Formatted at 2026-09-24 07:16:25 on dist-test-slave-zx46"
server_key: "c885f5cce2e094311d487f69eedf5cad"
server_key_iv: "39e07a2dec9e8ec6ba4aa74d526d8318"
server_key_version: "encryptionkey@0"
I20260924 07:16:25.122453 18784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:25.149484 18784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:25.153306 18784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:25.153544 18784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:25.161046 18784 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:36191
I20260924 07:16:25.161383 18851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:36191 every 8 connection(s)
I20260924 07:16:25.162098 18784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:25.164815 18852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:25.167055 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18784
I20260924 07:16:25.167191 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:25.167438 15870 external_mini_cluster.cc:1428] Setting key e2afdfe6c8cabe1b37625543c4f57687
I20260924 07:16:25.171289 18852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d: Bootstrap starting.
I20260924 07:16:25.173940 18852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:25.174849 18852 log.cc:824] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:25.176774 18852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d: No bootstrap required, opened a new log
Sep 24 07:16:25 dist-test-slave-zx46 krb5kdc[18771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:25.181213 18852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36191 } }
I20260924 07:16:25.181496 18852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:25.181560 18852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ad5ab2e38034e8eb81966dd92513b5d, State: Initialized, Role: FOLLOWER
I20260924 07:16:25.182077 18852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36191 } }
I20260924 07:16:25.182240 18852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:25.182305 18852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:25.182389 18852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:25.183527 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:25.168847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.15.127.190:36191)
0924 07:16:25.169354 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:25.169375 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:25.169417 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:16:25.170229 (+ 812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:25.170253 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:25.170682 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:25.171044 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:25.171800 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:25.173318 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:25.174269 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:25.174972 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:25.175133 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:25.177667 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:25.177673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:25.177689 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:25.177720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:25.180350 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:25.180798 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:25.180804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:25.180810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:25.180950 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:25.181243 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:25.181246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:25.181248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:25.181761 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:25.181950 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:25.182218 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 07:16:25.182539 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":191,"threads_started":1}
I20260924 07:16:25.183499 18852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36191 } }
I20260924 07:16:25.183866 18852 leader_election.cc:302] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ad5ab2e38034e8eb81966dd92513b5d; no voters:
I20260924 07:16:25.184211 18852 leader_election.cc:288] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:25.184326 18857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:25.184569 18857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [term 1 LEADER]: Becoming Leader. State: Replica: 1ad5ab2e38034e8eb81966dd92513b5d, State: Running, Role: LEADER
I20260924 07:16:25.184875 18857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36191 } }
I20260924 07:16:25.185462 18852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:25.186725 18858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36191 } } }
I20260924 07:16:25.186868 18858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:25.186966 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:16:25.187188 18859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ad5ab2e38034e8eb81966dd92513b5d. Latest consensus state: current_term: 1 leader_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ad5ab2e38034e8eb81966dd92513b5d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36191 } } }
I20260924 07:16:25.187270 18859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d [sys.catalog]: This master's current role is: LEADER
W20260924 07:16:25.190558 18872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:16:25.190689 18872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:16:25.190778 18867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:25.191646 18867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:25.198132 18867 catalog_manager.cc:1347] Generated new cluster ID: 47dd9da4c750410c90a54d6c3b691cfe
I20260924 07:16:25.198211 18867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:25.205065 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:25.200209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.15.127.190:36191)
0924 07:16:25.200329 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:25.200334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:25.200574 (+ 240us) server_negotiation.cc:407] Connection header received
0924 07:16:25.200740 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:25.200743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:25.200793 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:25.200879 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:25.201465 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:25.202004 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:25.202850 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:25.203105 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:25.203218 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:25.204084 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:25.204087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:25.204090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:25.204528 (+ 438us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:25.204557 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:25.204573 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:25.204676 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:25.204802 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:16:25.204863 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:16:25.208210 18867 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:25.209187 18867 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:25.215394 18867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ad5ab2e38034e8eb81966dd92513b5d: Generated new TSK 0
I20260924 07:16:25.216118 18867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:16:25.221063 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18784
2026-09-24T07:16:25Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1413 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:25 dist-test-slave-zx46 krb5kdc[18885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:25 dist-test-slave-zx46 krb5kdc[18885](info): set up 2 sockets
Sep 24 07:16:25 dist-test-slave-zx46 krb5kdc[18885](info): commencing operation
krb5kdc: starting...
W20260924 07:16:26.260990 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:26 dist-test-slave-zx46 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:26.396378 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:45027
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:26.507262 18898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:26.507550 18898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:26.507591 18898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:26.511616 18898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:26.511677 18898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:26.511698 18898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:26.511713 18898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:26.511729 18898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:26.517096 18898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:45027
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:26.518153 18898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:26.519152 18898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:26.527792 18906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:26.527828 18903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:26.528065 18898 server_base.cc:1034] running on GCE node
W20260924 07:16:26.528254 18904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:26.529081 18898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:26.530318 18898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:26.531477 18898 hybrid_clock.cc:648] HybridClock initialized: now 1790234186531447 us; error 46 us; skew 500 ppm
Sep 24 07:16:26 dist-test-slave-zx46 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234186, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:26.534379 18898 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:26.535388 18898 webserver.cc:460] Webserver started at http://127.15.127.190:45205/ using document root <none> and password file <none>
I20260924 07:16:26.535971 18898 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:26.536067 18898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:26.536278 18898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:26.538417 18898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "368e4e28c93f466591b34a0f821dd4a2"
format_stamp: "Formatted at 2026-09-24 07:16:26 on dist-test-slave-zx46"
server_key: "8aa9ae7ed13810b62a11684741bf6ee0"
server_key_iv: "ae6328d0ddd7a46b4d36f43d748f9a51"
server_key_version: "encryptionkey@0"
I20260924 07:16:26.538998 18898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "368e4e28c93f466591b34a0f821dd4a2"
format_stamp: "Formatted at 2026-09-24 07:16:26 on dist-test-slave-zx46"
server_key: "8aa9ae7ed13810b62a11684741bf6ee0"
server_key_iv: "ae6328d0ddd7a46b4d36f43d748f9a51"
server_key_version: "encryptionkey@0"
I20260924 07:16:26.542791 18898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 07:16:26.545300 18913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:26.546550 18898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:16:26.546715 18898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "368e4e28c93f466591b34a0f821dd4a2"
format_stamp: "Formatted at 2026-09-24 07:16:26 on dist-test-slave-zx46"
server_key: "8aa9ae7ed13810b62a11684741bf6ee0"
server_key_iv: "ae6328d0ddd7a46b4d36f43d748f9a51"
server_key_version: "encryptionkey@0"
I20260924 07:16:26.546833 18898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:26.563200 18898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:26.566205 18898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:26.566422 18898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:26.573609 18898 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:45027
I20260924 07:16:26.573884 18965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:45027 every 8 connection(s)
I20260924 07:16:26.574652 18898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:26.577654 18966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:26.582580 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18898
I20260924 07:16:26.582731 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:26.582993 15870 external_mini_cluster.cc:1428] Setting key a0838454fb123a9c003b426d6b9544ca
I20260924 07:16:26.583204 18966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2: Bootstrap starting.
I20260924 07:16:26.585875 18966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:26.586764 18966 log.cc:824] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:26.588706 18966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2: No bootstrap required, opened a new log
Sep 24 07:16:26 dist-test-slave-zx46 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:26.592485 18966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e4e28c93f466591b34a0f821dd4a2" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45027 } }
I20260924 07:16:26.592734 18966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:26.592814 18966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 368e4e28c93f466591b34a0f821dd4a2, State: Initialized, Role: FOLLOWER
I20260924 07:16:26.593325 18966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e4e28c93f466591b34a0f821dd4a2" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45027 } }
I20260924 07:16:26.593492 18966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:26.593580 18966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:26.593710 18966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:26.594635 18966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e4e28c93f466591b34a0f821dd4a2" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45027 } }
I20260924 07:16:26.594995 18966 leader_election.cc:302] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 368e4e28c93f466591b34a0f821dd4a2; no voters:
I20260924 07:16:26.595405 18966 leader_election.cc:288] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:26.595469 18971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:26.595672 18971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [term 1 LEADER]: Becoming Leader. State: Replica: 368e4e28c93f466591b34a0f821dd4a2, State: Running, Role: LEADER
I20260924 07:16:26.595973 18971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e4e28c93f466591b34a0f821dd4a2" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45027 } }
I20260924 07:16:26.596752 18966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:26.597209 18971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 368e4e28c93f466591b34a0f821dd4a2. Latest consensus state: current_term: 1 leader_uuid: "368e4e28c93f466591b34a0f821dd4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e4e28c93f466591b34a0f821dd4a2" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45027 } } }
I20260924 07:16:26.597323 18971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:26.597618 18972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "368e4e28c93f466591b34a0f821dd4a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "368e4e28c93f466591b34a0f821dd4a2" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45027 } } }
I20260924 07:16:26.597747 18972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:26.598438 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:26.584414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32798 (local address 127.15.127.190:45027)
0924 07:16:26.584846 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:26.584866 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:26.584909 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:16:26.585780 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:26.585807 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:26.586277 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:26.586663 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:26.587412 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:26.588410 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:26.589393 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:26.589874 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:26.590048 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:26.592756 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:26.592765 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:26.592790 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:26.592839 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:26.594859 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:26.595404 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:26.595411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:26.595417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:26.595570 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:26.595860 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:26.595865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:26.595867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:26.596427 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:26.596638 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:26.596907 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 07:16:26.597476 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":117,"threads_started":1}
I20260924 07:16:26.599563 18975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:26.602998 18975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:26.608574 18975 catalog_manager.cc:1347] Generated new cluster ID: 356fbbc5ea3345db88acd6ed7656ce4c
I20260924 07:16:26.608729 18975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:26.634402 18975 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:26.635789 18975 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:26.645762 18975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 368e4e28c93f466591b34a0f821dd4a2: Generated new TSK 0
I20260924 07:16:26.646514 18975 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:26.681437 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45027
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:26.794932 18992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:26.795217 18992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:26.795260 18992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:26.799643 18992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:26.799707 18992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:26.799765 18992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:26.805712 18992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:26.806824 18992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:26.807858 18992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:26.817443 18997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:26.817619 18992 server_base.cc:1034] running on GCE node
W20260924 07:16:26.817564 19000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:26.817539 18998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:26.818867 18992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:26.819502 18992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:26.820672 18992 hybrid_clock.cc:648] HybridClock initialized: now 1790234186820658 us; error 27 us; skew 500 ppm
Sep 24 07:16:26 dist-test-slave-zx46 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234186, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:26.826872 18992 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:26.827862 18992 webserver.cc:460] Webserver started at http://127.15.127.129:33249/ using document root <none> and password file <none>
I20260924 07:16:26.828466 18992 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:26.828533 18992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:26.828758 18992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:26.830960 18992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "cb5ec206c0814f9bae30221ce4105079"
format_stamp: "Formatted at 2026-09-24 07:16:26 on dist-test-slave-zx46"
server_key: "7dbed7692f7a985e2adb913d5725a0a0"
server_key_iv: "0aa8c546b42cab7b117687922fe84c47"
server_key_version: "encryptionkey@0"
I20260924 07:16:26.831554 18992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "cb5ec206c0814f9bae30221ce4105079"
format_stamp: "Formatted at 2026-09-24 07:16:26 on dist-test-slave-zx46"
server_key: "7dbed7692f7a985e2adb913d5725a0a0"
server_key_iv: "0aa8c546b42cab7b117687922fe84c47"
server_key_version: "encryptionkey@0"
I20260924 07:16:26.835528 18992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 07:16:26.838085 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:26.839398 18992 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:26.839560 18992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "cb5ec206c0814f9bae30221ce4105079"
format_stamp: "Formatted at 2026-09-24 07:16:26 on dist-test-slave-zx46"
server_key: "7dbed7692f7a985e2adb913d5725a0a0"
server_key_iv: "0aa8c546b42cab7b117687922fe84c47"
server_key_version: "encryptionkey@0"
I20260924 07:16:26.839686 18992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:26.866039 18992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:26.869441 18992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:26.869665 18992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:26.870298 18992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:26.871170 18992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:26.871243 18992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:26.871316 18992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:26.871358 18992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:26.880757 18992 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:44857
I20260924 07:16:26.880780 19120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:44857 every 8 connection(s)
I20260924 07:16:26.881719 18992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:26.888214 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 18992
I20260924 07:16:26.888327 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:26.888666 15870 external_mini_cluster.cc:1428] Setting key 5794fd430550b27400f1bb177d0f8a8a
Sep 24 07:16:26 dist-test-slave-zx46 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234186, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:26.896441 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:26.883622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:33109 (local address 127.15.127.190:45027)
0924 07:16:26.883769 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:26.883772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:26.884651 (+ 879us) server_negotiation.cc:407] Connection header received
0924 07:16:26.886114 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:26.886118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:26.886176 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:26.886260 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:26.887492 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:26.888063 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:26.889577 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:26.889821 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:26.889962 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:26.892752 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:26.892755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:26.892760 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:26.892790 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:26.894342 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:26.894874 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:26.894879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:26.894881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:26.894965 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:26.895324 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:26.895328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:26.895329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:26.895520 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:26.895629 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:26.896149 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 07:16:26.896274 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:16:26.897508 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:26.884040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45027 (local address 127.15.127.129:33109)
0924 07:16:26.884493 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:26.884537 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:26.885879 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:26.886444 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:26.886452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:26.886835 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:26.887336 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:26.887352 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:26.888199 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:26.888202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:26.889440 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:26.889452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:26.890045 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:26.890049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:26.890257 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:26.890910 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:26.890926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:26.892546 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:26.894460 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:26.894466 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:26.894484 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:26.894767 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:26.895063 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:26.895066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:26.895068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:26.895221 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:26.895632 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:26.895641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:26.895903 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 07:16:26.896105 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 07:16:26.896485 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:26.898774 19121 heartbeater.cc:347] Connected to a master server at 127.15.127.190:45027
I20260924 07:16:26.899044 19121 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:26.899607 19121 heartbeater.cc:511] Master 127.15.127.190:45027 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:16:26.901176 18930 ts_manager.cc:194] Registered new tserver with Master: cb5ec206c0814f9bae30221ce4105079 (127.15.127.129:44857)
I20260924 07:16:26.902832 18930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:33109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:26.916332 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45027
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:27.027917 19126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:27.028239 19126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:27.028324 19126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:27.032397 19126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:27.032491 19126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:27.032579 19126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:27.038009 19126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:27.039127 19126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:27.040272 19126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:27.050395 19132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:27.050349 19131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:27.050432 19126 server_base.cc:1034] running on GCE node
W20260924 07:16:27.050343 19134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:27.051051 19126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:27.051713 19126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:27.052896 19126 hybrid_clock.cc:648] HybridClock initialized: now 1790234187052869 us; error 47 us; skew 500 ppm
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234187, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:27.055593 19126 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:27.056474 19126 webserver.cc:460] Webserver started at http://127.15.127.130:38465/ using document root <none> and password file <none>
I20260924 07:16:27.057119 19126 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:27.057217 19126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:27.057451 19126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:27.059502 19126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "5c69d891abcc46a08ac68e7f446c6331"
format_stamp: "Formatted at 2026-09-24 07:16:27 on dist-test-slave-zx46"
server_key: "efd06ce6a593fdefbd724f1d7868bf27"
server_key_iv: "ef9e71050e6ef6f7b8d8f69d44621d59"
server_key_version: "encryptionkey@0"
I20260924 07:16:27.060062 19126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "5c69d891abcc46a08ac68e7f446c6331"
format_stamp: "Formatted at 2026-09-24 07:16:27 on dist-test-slave-zx46"
server_key: "efd06ce6a593fdefbd724f1d7868bf27"
server_key_iv: "ef9e71050e6ef6f7b8d8f69d44621d59"
server_key_version: "encryptionkey@0"
I20260924 07:16:27.063779 19126 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:16:27.066262 19141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:27.067528 19126 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:27.067651 19126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "5c69d891abcc46a08ac68e7f446c6331"
format_stamp: "Formatted at 2026-09-24 07:16:27 on dist-test-slave-zx46"
server_key: "efd06ce6a593fdefbd724f1d7868bf27"
server_key_iv: "ef9e71050e6ef6f7b8d8f69d44621d59"
server_key_version: "encryptionkey@0"
I20260924 07:16:27.067759 19126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:27.093977 19126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:27.096966 19126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:27.097247 19126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:27.097932 19126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:27.098843 19126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:27.098891 19126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:27.098960 19126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:27.098999 19126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:27.107825 19126 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:41395
I20260924 07:16:27.107844 19254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:41395 every 8 connection(s)
I20260924 07:16:27.108762 19126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:27.113199 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19126
I20260924 07:16:27.113339 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:27.113608 15870 external_mini_cluster.cc:1428] Setting key c5fa46cc8fb9d7c5975865375242950d
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234187, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:27.123912 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:27.110677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:44169 (local address 127.15.127.190:45027)
0924 07:16:27.110823 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:27.110827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:27.111654 (+ 827us) server_negotiation.cc:407] Connection header received
0924 07:16:27.113061 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:27.113065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:27.113132 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:27.113215 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:27.114663 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.115365 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.116574 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.116862 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.116957 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:27.120115 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:27.120118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:27.120120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:27.120144 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:27.121778 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:27.122307 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:27.122313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:27.122317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:27.122399 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:27.122740 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:27.122745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:27.122746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:27.122931 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:27.123052 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:27.123624 (+ 572us) server_negotiation.cc:299] Negotiation successful
0924 07:16:27.123749 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:27.124815 19257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:27.111050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45027 (local address 127.15.127.130:44169)
0924 07:16:27.111506 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:27.111548 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:27.112819 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:27.113402 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:27.113414 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:27.113843 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:27.114508 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:27.114529 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.115495 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.115497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:27.116411 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:27.116421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.117057 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.117064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:27.117296 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:27.118114 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:27.118131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:27.119914 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:27.121906 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:27.121912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:27.121930 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:27.122202 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:27.122497 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:27.122501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:27.122503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:27.122646 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:27.123062 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:27.123071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:27.123369 (+ 298us) client_negotiation.cc:769] Sending connection context
0924 07:16:27.123584 (+ 215us) client_negotiation.cc:240] Negotiation successful
0924 07:16:27.123924 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:27.126111 19255 heartbeater.cc:347] Connected to a master server at 127.15.127.190:45027
I20260924 07:16:27.126389 19255 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:27.126914 19255 heartbeater.cc:511] Master 127.15.127.190:45027 requested a full tablet report, sending...
I20260924 07:16:27.128034 18930 ts_manager.cc:194] Registered new tserver with Master: 5c69d891abcc46a08ac68e7f446c6331 (127.15.127.130:41395)
I20260924 07:16:27.128849 18930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:44169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:27.142146 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45027
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:38361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:27.252506 19260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:27.252836 19260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:27.252918 19260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:27.257134 19260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:27.257217 19260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:27.257309 19260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:27.262723 19260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:38361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:27.263823 19260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:27.264868 19260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:16:27.274746 19260 server_base.cc:1034] running on GCE node
W20260924 07:16:27.274725 19268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:27.274703 19266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:27.274726 19265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:27.275552 19260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:27.276235 19260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:27.277449 19260 hybrid_clock.cc:648] HybridClock initialized: now 1790234187277435 us; error 32 us; skew 500 ppm
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234187, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:27.280354 19260 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:27.281313 19260 webserver.cc:460] Webserver started at http://127.15.127.131:33075/ using document root <none> and password file <none>
I20260924 07:16:27.281875 19260 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:27.281936 19260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:27.282117 19260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:27.284071 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "170e55ef006a453ab1aefe5c6693db81"
format_stamp: "Formatted at 2026-09-24 07:16:27 on dist-test-slave-zx46"
server_key: "1d967680efce74258e09b775303a31d1"
server_key_iv: "0149237217e06efff9ad539d68a6960e"
server_key_version: "encryptionkey@0"
I20260924 07:16:27.284570 19260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "170e55ef006a453ab1aefe5c6693db81"
format_stamp: "Formatted at 2026-09-24 07:16:27 on dist-test-slave-zx46"
server_key: "1d967680efce74258e09b775303a31d1"
server_key_iv: "0149237217e06efff9ad539d68a6960e"
server_key_version: "encryptionkey@0"
I20260924 07:16:27.288151 19260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:16:27.290504 19275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:27.291738 19260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:27.291872 19260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "170e55ef006a453ab1aefe5c6693db81"
format_stamp: "Formatted at 2026-09-24 07:16:27 on dist-test-slave-zx46"
server_key: "1d967680efce74258e09b775303a31d1"
server_key_iv: "0149237217e06efff9ad539d68a6960e"
server_key_version: "encryptionkey@0"
I20260924 07:16:27.291981 19260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:27.312908 19260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:27.315979 19260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:27.316193 19260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:27.316818 19260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:27.317678 19260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:27.317732 19260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:27.317771 19260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:27.317787 19260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:27.326889 19260 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:38711
I20260924 07:16:27.326910 19388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:38711 every 8 connection(s)
I20260924 07:16:27.327797 19260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:27.328649 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19260
I20260924 07:16:27.328779 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:27.329157 15870 external_mini_cluster.cc:1428] Setting key 37bc5caac5e45e0fa4239d5f1a101bfb
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234187, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:27.342154 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:27.329848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:53775 (local address 127.15.127.190:45027)
0924 07:16:27.329992 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:27.329999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:27.330965 (+ 966us) server_negotiation.cc:407] Connection header received
0924 07:16:27.332405 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:27.332409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:27.332472 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:27.332549 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:27.333819 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.334366 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.335528 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.335767 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.335877 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:27.338560 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:27.338564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:27.338567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:27.338610 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:27.340033 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:27.340592 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:27.340597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:27.340598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:27.340684 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:27.341039 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:27.341043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:27.341045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:27.341241 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:27.341348 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:27.341860 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 07:16:27.341985 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:16:27.343250 19391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:27.330323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45027 (local address 127.15.127.131:53775)
0924 07:16:27.330817 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:27.330858 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:27.332201 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:27.332734 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:27.332741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:27.333194 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:27.333680 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:27.333694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.334506 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.334514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:27.335397 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:27.335406 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.335915 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.335920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:27.336097 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:27.336763 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:27.336780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:27.338387 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:27.340191 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:27.340197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:27.340214 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:27.340473 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:27.340777 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:27.340781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:27.340784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:27.340926 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:27.341354 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:27.341363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:27.341618 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 07:16:27.341822 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 07:16:27.342175 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":138,"threads_started":1}
I20260924 07:16:27.344300 19389 heartbeater.cc:347] Connected to a master server at 127.15.127.190:45027
I20260924 07:16:27.344597 19389 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:27.345182 19389 heartbeater.cc:511] Master 127.15.127.190:45027 requested a full tablet report, sending...
I20260924 07:16:27.346300 18930 ts_manager.cc:194] Registered new tserver with Master: 170e55ef006a453ab1aefe5c6693db81 (127.15.127.131:38711)
I20260924 07:16:27.346879 18930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:53775
I20260924 07:16:27.355129 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[18885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234187, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[18885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234187, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:27.380411 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:27.371980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.15.127.190:45027)
0924 07:16:27.372156 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:27.372161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:27.372311 (+ 150us) server_negotiation.cc:407] Connection header received
0924 07:16:27.372483 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:27.372486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:27.372537 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:27.372641 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:27.373432 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.373954 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.374802 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:27.375022 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:27.375127 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:27.377553 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:27.377556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:27.377558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:27.377581 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:27.378944 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:27.379321 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:27.379323 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:27.379325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:27.379398 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:27.379628 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:27.379629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:27.379631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:27.379837 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:27.379939 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:27.380129 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 07:16:27.380221 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260924 07:16:27.380895 18930 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32800
I20260924 07:16:27.381754 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18992
I20260924 07:16:27.387624 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19126
I20260924 07:16:27.392889 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19260
I20260924 07:16:27.398242 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 18898
2026-09-24T07:16:27Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2178 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[19401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[19401](info): set up 2 sockets
Sep 24 07:16:27 dist-test-slave-zx46 krb5kdc[19401](info): commencing operation
krb5kdc: starting...
W20260924 07:16:28.433148 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:28 dist-test-slave-zx46 krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:28.564007 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:42179
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:42179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:28.676538 19414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:28.676831 19414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:28.676873 19414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:28.681108 19414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:28.681173 19414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:28.681192 19414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:28.681209 19414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:28.681226 19414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:28.686787 19414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:42179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:42179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:28.687952 19414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:28.689031 19414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:28.697679 19422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:28.697678 19420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:28.697705 19419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:28.697870 19414 server_base.cc:1034] running on GCE node
I20260924 07:16:28.698565 19414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:28.699752 19414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:28.700917 19414 hybrid_clock.cc:648] HybridClock initialized: now 1790234188700905 us; error 28 us; skew 500 ppm
Sep 24 07:16:28 dist-test-slave-zx46 krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234188, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:28.703744 19414 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:28.704659 19414 webserver.cc:460] Webserver started at http://127.15.127.190:34637/ using document root <none> and password file <none>
I20260924 07:16:28.705305 19414 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:28.705371 19414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:28.705613 19414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:28.707703 19414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "1638ce0d3abc4d5d967f0e997e0bf20b"
format_stamp: "Formatted at 2026-09-24 07:16:28 on dist-test-slave-zx46"
server_key: "bf92d564adfb6e2271be9b43bafefa11"
server_key_iv: "0c043137e1cc0b7170f5f387567a5e30"
server_key_version: "encryptionkey@0"
I20260924 07:16:28.708281 19414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "1638ce0d3abc4d5d967f0e997e0bf20b"
format_stamp: "Formatted at 2026-09-24 07:16:28 on dist-test-slave-zx46"
server_key: "bf92d564adfb6e2271be9b43bafefa11"
server_key_iv: "0c043137e1cc0b7170f5f387567a5e30"
server_key_version: "encryptionkey@0"
I20260924 07:16:28.712037 19414 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:16:28.714600 19429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:28.715919 19414 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:28.716075 19414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "1638ce0d3abc4d5d967f0e997e0bf20b"
format_stamp: "Formatted at 2026-09-24 07:16:28 on dist-test-slave-zx46"
server_key: "bf92d564adfb6e2271be9b43bafefa11"
server_key_iv: "0c043137e1cc0b7170f5f387567a5e30"
server_key_version: "encryptionkey@0"
I20260924 07:16:28.716205 19414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:28.748373 19414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:28.751489 19414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:28.751724 19414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:28.759239 19414 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:42179
I20260924 07:16:28.759258 19481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:42179 every 8 connection(s)
I20260924 07:16:28.760291 19414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:28.760967 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19414
I20260924 07:16:28.761147 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:28.761461 15870 external_mini_cluster.cc:1428] Setting key 95b8ff4e87d144085b94b16990d4d03b
I20260924 07:16:28.763655 19482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:28.770390 19482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b: Bootstrap starting.
Sep 24 07:16:28 dist-test-slave-zx46 krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:28.773239 19482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:28.774189 19482 log.cc:824] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:28.776238 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:28.762814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.15.127.190:42179)
0924 07:16:28.763212 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:28.763234 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:28.763281 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:16:28.763982 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:28.764000 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:28.764335 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:28.764629 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:28.765569 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:28.766575 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:28.767727 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:28.768181 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:28.768300 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:28.771238 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:28.771243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:28.771256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:28.771284 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:28.773073 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:28.773567 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:28.773572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:28.773577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:28.773707 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:28.774001 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:28.774004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:28.774005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:28.774479 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:28.774683 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:28.774975 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 07:16:28.775295 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":143,"threads_started":1}
I20260924 07:16:28.776577 19482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b: No bootstrap required, opened a new log
I20260924 07:16:28.780556 19482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42179 } }
I20260924 07:16:28.780793 19482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:28.780836 19482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1638ce0d3abc4d5d967f0e997e0bf20b, State: Initialized, Role: FOLLOWER
I20260924 07:16:28.781268 19482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42179 } }
I20260924 07:16:28.781388 19482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:28.781433 19482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:28.781503 19482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:28.782500 19482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42179 } }
I20260924 07:16:28.782799 19482 leader_election.cc:302] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1638ce0d3abc4d5d967f0e997e0bf20b; no voters:
I20260924 07:16:28.783066 19482 leader_election.cc:288] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:28.783215 19487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:28.783463 19487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [term 1 LEADER]: Becoming Leader. State: Replica: 1638ce0d3abc4d5d967f0e997e0bf20b, State: Running, Role: LEADER
I20260924 07:16:28.783779 19487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42179 } }
I20260924 07:16:28.784447 19482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:28.785722 19489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1638ce0d3abc4d5d967f0e997e0bf20b. Latest consensus state: current_term: 1 leader_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42179 } } }
I20260924 07:16:28.785735 19488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1638ce0d3abc4d5d967f0e997e0bf20b" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42179 } } }
I20260924 07:16:28.785858 19488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:28.785857 19489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:28.786244 19496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:28.788942 19496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:28.794452 19496 catalog_manager.cc:1347] Generated new cluster ID: b6dc214f849e4c548272f643e1b90588
I20260924 07:16:28.794535 19496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:28.819723 19496 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:28.821201 19496 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:28.833030 19496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1638ce0d3abc4d5d967f0e997e0bf20b: Generated new TSK 0
I20260924 07:16:28.833870 19496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:28.876310 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42179
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:29.019081 19508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:29.019480 19508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:29.019596 19508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:29.023907 19508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:29.024010 19508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:29.024106 19508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:29.029572 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:29.030694 19508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:29.031737 19508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:29.041551 19514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:29.041551 19513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:29.041626 19508 server_base.cc:1034] running on GCE node
W20260924 07:16:29.041543 19516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:29.042260 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:29.042936 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:29.044128 19508 hybrid_clock.cc:648] HybridClock initialized: now 1790234189044101 us; error 39 us; skew 500 ppm
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:29.047061 19508 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:29.047978 19508 webserver.cc:460] Webserver started at http://127.15.127.129:35157/ using document root <none> and password file <none>
I20260924 07:16:29.048579 19508 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:29.048676 19508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:29.048916 19508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:29.051062 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "33b4c57e63a9430ca01623fe0faad514"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "cc2920881edc1c67a3dda101e41e53f6"
server_key_iv: "ce0053e6ad687ff916fb7c84467bce25"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.051662 19508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "33b4c57e63a9430ca01623fe0faad514"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "cc2920881edc1c67a3dda101e41e53f6"
server_key_iv: "ce0053e6ad687ff916fb7c84467bce25"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.055564 19508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:16:29.058157 19523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.059669 19508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 07:16:29.059801 19508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "33b4c57e63a9430ca01623fe0faad514"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "cc2920881edc1c67a3dda101e41e53f6"
server_key_iv: "ce0053e6ad687ff916fb7c84467bce25"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.059950 19508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:29.088518 19508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:29.092136 19508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:29.092350 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:29.093042 19508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:29.093896 19508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:29.093943 19508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.094017 19508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:29.094295 19508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.103080 19508 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:45371
I20260924 07:16:29.103102 19636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:45371 every 8 connection(s)
I20260924 07:16:29.104007 19508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:29.104378 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19508
I20260924 07:16:29.104519 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:29.104854 15870 external_mini_cluster.cc:1428] Setting key e6030aa234f6364d89f78b2bce3479dc
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:29.123400 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.105914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:52585 (local address 127.15.127.190:42179)
0924 07:16:29.106046 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:29.106050 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:29.107136 (+ 1086us) server_negotiation.cc:407] Connection header received
0924 07:16:29.108894 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:29.108899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:29.108964 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:29.109082 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:29.110683 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.111311 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.113235 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.113470 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.113571 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.119194 (+ 5623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:29.119197 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:29.119199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:29.119228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:29.120959 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.121637 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.121642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.121644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.121731 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.122118 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.122121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.122123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.122341 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:29.122454 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:29.123128 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 07:16:29.123261 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:16:29.124953 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.106432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:42179 (local address 127.15.127.129:52585)
0924 07:16:29.106956 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:29.107015 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:29.108677 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:29.109256 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:29.109263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:29.109799 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:29.110498 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:29.110521 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.111454 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.111459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:29.113114 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:29.113127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.113664 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.113668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:29.113907 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.114571 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:29.114596 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:29.118961 (+ 4365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:29.121176 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:29.121184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:29.121204 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:29.121492 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:29.121831 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:29.121835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:29.121837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:29.122003 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:29.122453 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:29.122462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:29.122808 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 07:16:29.123065 (+ 257us) client_negotiation.cc:240] Negotiation successful
0924 07:16:29.123617 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260924 07:16:29.126675 19637 heartbeater.cc:347] Connected to a master server at 127.15.127.190:42179
I20260924 07:16:29.126955 19637 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:29.127560 19637 heartbeater.cc:511] Master 127.15.127.190:42179 requested a full tablet report, sending...
I20260924 07:16:29.129311 19442 ts_manager.cc:194] Registered new tserver with Master: 33b4c57e63a9430ca01623fe0faad514 (127.15.127.129:45371)
I20260924 07:16:29.130481 19442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:52585
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:29.135958 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42179
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:29.246879 19642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:29.247226 19642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:29.247303 19642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:29.251484 19642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:29.251600 19642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:29.251708 19642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:29.257207 19642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:29.258374 19642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:29.259459 19642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:29.269970 19650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:29.270056 19642 server_base.cc:1034] running on GCE node
W20260924 07:16:29.270227 19647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:29.269974 19648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:29.270733 19642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:29.271381 19642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:29.272570 19642 hybrid_clock.cc:648] HybridClock initialized: now 1790234189272554 us; error 29 us; skew 500 ppm
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:29.275738 19642 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:29.276664 19642 webserver.cc:460] Webserver started at http://127.15.127.130:44829/ using document root <none> and password file <none>
I20260924 07:16:29.277298 19642 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:29.277369 19642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:29.277554 19642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:29.279630 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "179df7183c9a44718c6c94673c99e1a7"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "687871d4ef5d3e682ed53c70d83a07a0"
server_key_iv: "ec20120dab1ecb083890caa11ee25de1"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.280154 19642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "179df7183c9a44718c6c94673c99e1a7"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "687871d4ef5d3e682ed53c70d83a07a0"
server_key_iv: "ec20120dab1ecb083890caa11ee25de1"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.284096 19642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260924 07:16:29.286935 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.288201 19642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:29.288324 19642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "179df7183c9a44718c6c94673c99e1a7"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "687871d4ef5d3e682ed53c70d83a07a0"
server_key_iv: "ec20120dab1ecb083890caa11ee25de1"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.288436 19642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:29.314152 19642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:29.317860 19642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:29.318089 19642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:29.318737 19642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:29.319675 19642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:29.319731 19642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.319796 19642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:29.319834 19642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.329185 19642 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:41777
I20260924 07:16:29.329209 19770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:41777 every 8 connection(s)
I20260924 07:16:29.330152 19642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:29.332845 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19642
I20260924 07:16:29.332986 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:29.333371 15870 external_mini_cluster.cc:1428] Setting key 42525bfec577144204ff165af2102d8a
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:29.345192 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.332069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:54349 (local address 127.15.127.190:42179)
0924 07:16:29.332213 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:29.332217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:29.333274 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 07:16:29.335102 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:29.335106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:29.335160 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:29.335249 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:29.336511 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.337101 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.338261 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.338488 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.338600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.341344 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:29.341347 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:29.341349 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:29.341372 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:29.342843 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.343344 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.343346 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.343348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.343420 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.343739 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.343741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.343742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.343950 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:29.344061 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:29.344785 (+ 724us) server_negotiation.cc:299] Negotiation successful
0924 07:16:29.344902 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:29.346062 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.332553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:42179 (local address 127.15.127.130:54349)
0924 07:16:29.333119 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:29.333175 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:29.334864 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:29.335408 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:29.335415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:29.335799 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:29.336371 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:29.336387 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.337225 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.337229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:29.338130 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:29.338139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.338640 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.338644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:29.338825 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.339450 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:29.339466 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:29.341166 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:29.342948 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:29.342954 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:29.342967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:29.343224 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:29.343507 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:29.343509 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:29.343511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:29.343656 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:29.344135 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:29.344141 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:29.344496 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 07:16:29.344705 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 07:16:29.344989 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":162,"threads_started":1}
I20260924 07:16:29.347353 19771 heartbeater.cc:347] Connected to a master server at 127.15.127.190:42179
I20260924 07:16:29.347651 19771 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:29.348273 19771 heartbeater.cc:511] Master 127.15.127.190:42179 requested a full tablet report, sending...
I20260924 07:16:29.349481 19442 ts_manager.cc:194] Registered new tserver with Master: 179df7183c9a44718c6c94673c99e1a7 (127.15.127.130:41777)
I20260924 07:16:29.350095 19442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:54349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:29.364606 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42179
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:29.478102 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:29.478459 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:29.478538 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:29.482863 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:29.482964 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:29.483058 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:29.489668 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:29.491221 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:29.492602 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:29.503367 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:29.503361 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:29.503361 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:29.503762 19776 server_base.cc:1034] running on GCE node
I20260924 07:16:29.504238 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:29.504995 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:29.506248 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790234189506229 us; error 31 us; skew 500 ppm
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:29.509292 19776 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:29.510226 19776 webserver.cc:460] Webserver started at http://127.15.127.131:34191/ using document root <none> and password file <none>
I20260924 07:16:29.510819 19776 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:29.510910 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:29.511147 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:29.513371 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "0ad3bee678134366bbec88f8e33bc9bb"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "67c0596149868e7502c688fc814d7c0e"
server_key_iv: "0a452b7464fa749b7e6b6e90e4d6a675"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.513972 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "0ad3bee678134366bbec88f8e33bc9bb"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "67c0596149868e7502c688fc814d7c0e"
server_key_iv: "0a452b7464fa749b7e6b6e90e4d6a675"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.517904 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:16:29.520651 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.522042 19776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:29.522270 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "0ad3bee678134366bbec88f8e33bc9bb"
format_stamp: "Formatted at 2026-09-24 07:16:29 on dist-test-slave-zx46"
server_key: "67c0596149868e7502c688fc814d7c0e"
server_key_iv: "0a452b7464fa749b7e6b6e90e4d6a675"
server_key_version: "encryptionkey@0"
I20260924 07:16:29.522403 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:29.541476 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:29.544591 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:29.544816 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:29.545485 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:29.546334 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:29.546379 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.546440 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:29.546474 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:29.555573 19776 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:41047
I20260924 07:16:29.555651 19904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:41047 every 8 connection(s)
I20260924 07:16:29.556624 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:29.561611 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19776
I20260924 07:16:29.561784 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:29.562071 15870 external_mini_cluster.cc:1428] Setting key 4dea734b63aca45f28eca2d6ab675624
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:29.571074 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.558798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:52001 (local address 127.15.127.190:42179)
0924 07:16:29.558949 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:29.558953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:29.559964 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 07:16:29.561466 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:29.561470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:29.561527 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:29.561635 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:29.563023 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.563560 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.564873 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.565155 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.565270 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.567686 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:29.567690 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:29.567693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:29.567725 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:29.568971 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.569526 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.569530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.569532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.569614 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.569962 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.569965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.569967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.570158 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:29.570267 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:29.570784 (+ 517us) server_negotiation.cc:299] Negotiation successful
0924 07:16:29.570909 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:16:29.571940 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.559290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:42179 (local address 127.15.127.131:52001)
0924 07:16:29.559809 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:29.559855 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:29.561246 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:29.561865 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:29.561877 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:29.562322 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:29.562882 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:29.562899 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.563694 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.563698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:29.564748 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:29.564758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.565311 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.565315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:29.565495 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.566056 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:29.566073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:29.567512 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:29.569128 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:29.569135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:29.569153 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:29.569421 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:29.569718 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:29.569721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:29.569723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:29.569860 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:29.570275 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:29.570285 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:29.570541 (+ 256us) client_negotiation.cc:769] Sending connection context
0924 07:16:29.570746 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 07:16:29.571089 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":143,"threads_started":1}
I20260924 07:16:29.573271 19905 heartbeater.cc:347] Connected to a master server at 127.15.127.190:42179
I20260924 07:16:29.573549 19905 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:29.574061 19905 heartbeater.cc:511] Master 127.15.127.190:42179 requested a full tablet report, sending...
I20260924 07:16:29.575222 19446 ts_manager.cc:194] Registered new tserver with Master: 0ad3bee678134366bbec88f8e33bc9bb (127.15.127.131:41047)
I20260924 07:16:29.575915 19446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:52001
I20260924 07:16:29.576095 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:29.602716 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.592893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45398 (local address 127.15.127.129:45371)
0924 07:16:29.593248 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:29.593264 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:29.593294 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:16:29.593357 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:29.593366 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:29.593592 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:29.593757 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:29.594437 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.595253 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.596093 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.596481 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.596603 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.599224 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:29.599235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:29.599249 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:29.599277 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:29.600836 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.601315 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.601328 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.601331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.601409 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.601701 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.601704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.601706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.601965 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:29.602220 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:29.602372 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:16:29.602530 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:29.614529 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:29.606011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.15.127.190:42179)
0924 07:16:29.606180 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:29.606184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:29.606199 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:29.606299 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:29.606301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:29.606342 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:29.606411 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:29.606993 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.607509 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.608258 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:29.608464 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:29.608541 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:29.611126 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:29.611129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:29.611132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:29.611159 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:29.612671 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.613223 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.613226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.613227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.613304 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:29.613596 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:29.613599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:29.613600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:29.613862 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:29.614010 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:29.614215 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:16:29.614318 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 07:16:29.615005 19446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59920
I20260924 07:16:29.615721 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19508
I20260924 07:16:29.621790 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19642
I20260924 07:16:29.627789 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19776
I20260924 07:16:29.633172 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19414
2026-09-24T07:16:29Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2235 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19917](info): set up 2 sockets
Sep 24 07:16:29 dist-test-slave-zx46 krb5kdc[19917](info): commencing operation
krb5kdc: starting...
W20260924 07:16:30.677148 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:30 dist-test-slave-zx46 krb5kdc[19917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:30.803836 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:34477
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:34477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:30.915277 19930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:30.915583 19930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:30.915629 19930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:30.919862 19930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:30.919925 19930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:30.919945 19930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:30.919962 19930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:30.920001 19930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:30.925452 19930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:34477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:34477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:30.926645 19930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:30.927704 19930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:30.936206 19936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:30.936213 19935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:30.936251 19930 server_base.cc:1034] running on GCE node
W20260924 07:16:30.936213 19938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:30.937063 19930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:30.938215 19930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:30.939400 19930 hybrid_clock.cc:648] HybridClock initialized: now 1790234190939367 us; error 47 us; skew 500 ppm
Sep 24 07:16:30 dist-test-slave-zx46 krb5kdc[19917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234190, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:30.942230 19930 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:30.942972 19930 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:30.943037 19930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:30.943200 19930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:30.945271 19930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "7828d9a3d12b485ab18bb48f1618751f"
format_stamp: "Formatted at 2026-09-24 07:16:30 on dist-test-slave-zx46"
server_key: "ee3028658a01f53843f589d77970d056"
server_key_iv: "f9e574e0add8e6ccad27ed9eb0dbf366"
server_key_version: "encryptionkey@0"
I20260924 07:16:30.945782 19930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "7828d9a3d12b485ab18bb48f1618751f"
format_stamp: "Formatted at 2026-09-24 07:16:30 on dist-test-slave-zx46"
server_key: "ee3028658a01f53843f589d77970d056"
server_key_iv: "f9e574e0add8e6ccad27ed9eb0dbf366"
server_key_version: "encryptionkey@0"
I20260924 07:16:30.949470 19930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:30.951835 19944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:30.953260 19930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:30.953394 19930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "7828d9a3d12b485ab18bb48f1618751f"
format_stamp: "Formatted at 2026-09-24 07:16:30 on dist-test-slave-zx46"
server_key: "ee3028658a01f53843f589d77970d056"
server_key_iv: "f9e574e0add8e6ccad27ed9eb0dbf366"
server_key_version: "encryptionkey@0"
I20260924 07:16:30.953502 19930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:30.980252 19930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:30.983522 19930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:30.983691 19930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:30.990993 19930 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:34477
I20260924 07:16:30.991272 19996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:34477 every 8 connection(s)
I20260924 07:16:30.991742 19930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:30.994524 19997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:31.000137 19997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f: Bootstrap starting.
I20260924 07:16:31.000663 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 19930
I20260924 07:16:31.000789 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:31.001098 15870 external_mini_cluster.cc:1428] Setting key c41a024fa02bdf1269dfa3fd535afa7c
I20260924 07:16:31.003062 19997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:31.003849 19997 log.cc:824] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:31.006166 19997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f: No bootstrap required, opened a new log
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:31.010942 19997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7828d9a3d12b485ab18bb48f1618751f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34477 } }
I20260924 07:16:31.011286 19997 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:31.011364 19997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7828d9a3d12b485ab18bb48f1618751f, State: Initialized, Role: FOLLOWER
I20260924 07:16:31.011803 19997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7828d9a3d12b485ab18bb48f1618751f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34477 } }
I20260924 07:16:31.011951 19997 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:31.012044 19997 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:31.012169 19997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:31.013165 19997 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7828d9a3d12b485ab18bb48f1618751f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34477 } }
I20260924 07:16:31.013521 19997 leader_election.cc:302] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7828d9a3d12b485ab18bb48f1618751f; no voters:
I20260924 07:16:31.013883 19997 leader_election.cc:288] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:31.013985 20002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:31.014161 20002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [term 1 LEADER]: Becoming Leader. State: Replica: 7828d9a3d12b485ab18bb48f1618751f, State: Running, Role: LEADER
I20260924 07:16:31.014525 20002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7828d9a3d12b485ab18bb48f1618751f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34477 } }
I20260924 07:16:31.015264 19997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:31.016309 20003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7828d9a3d12b485ab18bb48f1618751f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7828d9a3d12b485ab18bb48f1618751f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34477 } } }
I20260924 07:16:31.016448 20003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:31.016852 20004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7828d9a3d12b485ab18bb48f1618751f. Latest consensus state: current_term: 1 leader_uuid: "7828d9a3d12b485ab18bb48f1618751f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7828d9a3d12b485ab18bb48f1618751f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 34477 } } }
I20260924 07:16:31.017023 20004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:31.017313 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.002515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.15.127.190:34477)
0924 07:16:31.003025 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.003041 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.003079 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:16:31.003812 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.003829 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.004275 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.004674 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:31.005566 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.006679 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.007875 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.008386 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.008509 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.011607 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:31.011614 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:31.011635 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:31.011673 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:31.013772 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.014300 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.014306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.014312 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.014451 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.014783 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.014786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.014788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.015333 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:31.015477 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.015946 (+ 469us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.016279 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":139,"threads_started":1}
I20260924 07:16:31.017812 20011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:31.020473 20011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:31.025799 20011 catalog_manager.cc:1347] Generated new cluster ID: b2dcf87350c747a7acebce553d5b1efe
I20260924 07:16:31.025876 20011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:31.035822 20011 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:31.037154 20011 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:31.042508 20011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f: Generated new TSK 0
I20260924 07:16:31.043359 20011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:31.080068 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34477
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:31.194785 20023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:31.195071 20023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:31.195118 20023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:31.199259 20023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:31.199332 20023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:31.199396 20023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:31.204639 20023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:31.205838 20023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:31.206840 20023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:31.216495 20028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:31.216487 20029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:31.216487 20031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:31.216661 20023 server_base.cc:1034] running on GCE node
I20260924 07:16:31.217345 20023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:31.217963 20023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:31.219123 20023 hybrid_clock.cc:648] HybridClock initialized: now 1790234191219086 us; error 49 us; skew 500 ppm
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234191, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:31.222000 20023 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:31.222802 20023 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:31.222898 20023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:31.223138 20023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:31.225270 20023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "f274715d511743c3b765da7883786fd0"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "398091756cc44deef54e27c643fcf7c6"
server_key_iv: "7e1916ec2db220360231daf8cbd743da"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.225848 20023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "f274715d511743c3b765da7883786fd0"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "398091756cc44deef54e27c643fcf7c6"
server_key_iv: "7e1916ec2db220360231daf8cbd743da"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.229558 20023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:31.232023 20037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.233366 20023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:31.233500 20023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "f274715d511743c3b765da7883786fd0"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "398091756cc44deef54e27c643fcf7c6"
server_key_iv: "7e1916ec2db220360231daf8cbd743da"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.233624 20023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:31.252460 20023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:31.255911 20023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:31.256137 20023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:31.256865 20023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:31.257814 20023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:31.257898 20023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.257954 20023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:31.257977 20023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.267318 20023 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:34749
I20260924 07:16:31.267335 20150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:34749 every 8 connection(s)
I20260924 07:16:31.267982 20023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:31.276942 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20023
I20260924 07:16:31.277097 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:31.277460 15870 external_mini_cluster.cc:1428] Setting key 13aabb5f46ee67c4df640dec69d6ddec
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234191, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:31.282640 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.269825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60769 (local address 127.15.127.190:34477)
0924 07:16:31.269956 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.269960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.270796 (+ 836us) server_negotiation.cc:407] Connection header received
0924 07:16:31.272125 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.272129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.272190 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.272270 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:31.273622 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.274165 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.275335 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.275581 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.275694 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.278621 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:31.278625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:31.278627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:31.278652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:31.280392 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.280986 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.280989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.280991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.281144 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.281503 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.281508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.281510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.281713 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:31.281822 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.282346 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.282477 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:16:31.283515 20153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.270176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34477 (local address 127.15.127.129:60769)
0924 07:16:31.270638 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.270681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.271888 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.272449 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.272456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.272885 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:31.273465 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.273480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.274300 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.274310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.275199 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.275208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.275750 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.275754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.275925 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.276519 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:31.276536 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:31.278436 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:31.280520 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:31.280525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:31.280540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:31.280869 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:31.281244 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:31.281249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:31.281251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:31.281405 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:31.281821 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:31.281830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:31.282098 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.282297 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.282655 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":115,"threads_started":1}
I20260924 07:16:31.284969 20151 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34477
I20260924 07:16:31.285272 20151 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:31.285763 20151 heartbeater.cc:511] Master 127.15.127.190:34477 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:31.287402 19961 ts_manager.cc:194] Registered new tserver with Master: f274715d511743c3b765da7883786fd0 (127.15.127.129:34749)
I20260924 07:16:31.288642 19961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:60769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:31.305546 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34477
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:31.415222 20156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:31.415491 20156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:31.415536 20156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:31.419808 20156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:31.419878 20156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:31.419942 20156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:31.425400 20156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:31.426512 20156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:31.427542 20156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:31.437180 20164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:31.437202 20161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:31.437225 20162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:31.437497 20156 server_base.cc:1034] running on GCE node
I20260924 07:16:31.438107 20156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:31.438786 20156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:31.439945 20156 hybrid_clock.cc:648] HybridClock initialized: now 1790234191439930 us; error 29 us; skew 500 ppm
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234191, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:31.442795 20156 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:31.443547 20156 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:31.443608 20156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:31.443825 20156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:31.445916 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "dc707ccccba34689870bd68d4df7ad0f"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "f857db87d088e0173d37c8e07b977f22"
server_key_iv: "e443d0ef60315f451bf223f4db55b22f"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.446475 20156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "dc707ccccba34689870bd68d4df7ad0f"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "f857db87d088e0173d37c8e07b977f22"
server_key_iv: "e443d0ef60315f451bf223f4db55b22f"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.450142 20156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 07:16:31.452474 20170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.453786 20156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:16:31.453928 20156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "dc707ccccba34689870bd68d4df7ad0f"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "f857db87d088e0173d37c8e07b977f22"
server_key_iv: "e443d0ef60315f451bf223f4db55b22f"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.454044 20156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:31.465306 20156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:31.468034 20156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:31.468241 20156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:31.468808 20156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:31.469694 20156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:31.469765 20156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.469836 20156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:31.469892 20156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.479136 20156 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:37747
I20260924 07:16:31.479192 20283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:37747 every 8 connection(s)
I20260924 07:16:31.479786 20156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:31.481720 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20156
I20260924 07:16:31.481848 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:31.482167 15870 external_mini_cluster.cc:1428] Setting key d27df1adfaa2ca3d171de2ca51bd5508
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234191, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:31.494997 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.481657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:48589 (local address 127.15.127.190:34477)
0924 07:16:31.481827 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.481833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.483019 (+ 1186us) server_negotiation.cc:407] Connection header received
0924 07:16:31.484426 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.484430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.484485 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.484564 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:31.485937 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.486471 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.487662 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.487972 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.488074 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.490947 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:31.490951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:31.490953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:31.490989 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:31.492528 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.493193 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.493197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.493199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.493284 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.493686 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.493689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.493691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.493913 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:31.494027 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.494689 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.494821 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:16:31.495870 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.482230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34477 (local address 127.15.127.130:48589)
0924 07:16:31.482861 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.482908 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.484206 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.484748 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.484756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.485231 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:31.485783 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.485797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.486604 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.486607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.487519 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.487528 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.488135 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.488139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.488316 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.488974 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:31.488991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:31.490761 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:31.492653 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:31.492662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:31.492683 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:31.493072 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:31.493380 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:31.493384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:31.493386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:31.493536 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:31.494025 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:31.494033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:31.494396 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.494626 (+ 230us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.494971 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":149,"threads_started":1}
I20260924 07:16:31.497289 20284 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34477
I20260924 07:16:31.497557 20284 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:31.498034 20284 heartbeater.cc:511] Master 127.15.127.190:34477 requested a full tablet report, sending...
I20260924 07:16:31.499192 19961 ts_manager.cc:194] Registered new tserver with Master: dc707ccccba34689870bd68d4df7ad0f (127.15.127.130:37747)
I20260924 07:16:31.500129 19961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:48589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:31.511447 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34477
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:31.628437 20289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:31.628727 20289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:31.628773 20289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:31.632951 20289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:31.633096 20289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:31.633220 20289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:31.639453 20289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:34477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:31.640550 20289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:31.641649 20289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:16:31.651012 20289 server_base.cc:1034] running on GCE node
W20260924 07:16:31.650974 20294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:31.651081 20295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:31.650993 20297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:31.651783 20289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:31.652434 20289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:31.653622 20289 hybrid_clock.cc:648] HybridClock initialized: now 1790234191653606 us; error 26 us; skew 500 ppm
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234191, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:31.656455 20289 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:31.657436 20289 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:31.657543 20289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:31.657769 20289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:31.660025 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "4937b3b6e2e6474c9e4af0a57f405ac0"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "6b9170a7530b9d1b7ddc58bf6ca6f91d"
server_key_iv: "b3277d8bf285509c7ed3fd4fb0dfce6a"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.660615 20289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "4937b3b6e2e6474c9e4af0a57f405ac0"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "6b9170a7530b9d1b7ddc58bf6ca6f91d"
server_key_iv: "b3277d8bf285509c7ed3fd4fb0dfce6a"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.664389 20289 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.004s
I20260924 07:16:31.666913 20303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.668161 20289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:31.668295 20289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "4937b3b6e2e6474c9e4af0a57f405ac0"
format_stamp: "Formatted at 2026-09-24 07:16:31 on dist-test-slave-zx46"
server_key: "6b9170a7530b9d1b7ddc58bf6ca6f91d"
server_key_iv: "b3277d8bf285509c7ed3fd4fb0dfce6a"
server_key_version: "encryptionkey@0"
I20260924 07:16:31.668407 20289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:31.677551 20289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:31.680619 20289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:31.680837 20289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:31.681469 20289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:31.682294 20289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:31.682366 20289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.682432 20289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:31.682488 20289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:31.691246 20289 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:40513
I20260924 07:16:31.691262 20416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:40513 every 8 connection(s)
I20260924 07:16:31.691848 20289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:31.697877 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20289
I20260924 07:16:31.698009 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:31.698300 15870 external_mini_cluster.cc:1428] Setting key 41bb5a8d7921b73157f67295468cd337
Sep 24 07:16:31 dist-test-slave-zx46 krb5kdc[19917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234191, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:31.705492 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.693641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:56455 (local address 127.15.127.190:34477)
0924 07:16:31.693785 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.693788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.694612 (+ 824us) server_negotiation.cc:407] Connection header received
0924 07:16:31.695872 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.695876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.695930 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.696039 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:31.697283 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.697819 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.699239 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.699563 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.699675 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.702373 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:31.702376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:31.702378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:31.702403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:31.703624 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.704083 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.704085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.704087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.704175 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.704465 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.704467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.704468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.704643 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:31.704718 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.705264 (+ 546us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.705368 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:16:31.706406 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.694013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:34477 (local address 127.15.127.131:56455)
0924 07:16:31.694461 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.694504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.695659 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.696230 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.696238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.696590 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:31.697137 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.697154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.697947 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.697950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.699071 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.699080 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.699738 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.699743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.699927 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.700650 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:31.700666 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:31.702215 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:31.703728 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:31.703733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:31.703746 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:31.703996 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:31.704253 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:31.704255 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:31.704256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:31.704388 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:31.704730 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:31.704735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:31.704982 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.705219 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.705490 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260924 07:16:31.707577 20417 heartbeater.cc:347] Connected to a master server at 127.15.127.190:34477
I20260924 07:16:31.707827 20417 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:31.708293 20417 heartbeater.cc:511] Master 127.15.127.190:34477 requested a full tablet report, sending...
I20260924 07:16:31.709370 19961 ts_manager.cc:194] Registered new tserver with Master: 4937b3b6e2e6474c9e4af0a57f405ac0 (127.15.127.131:40513)
I20260924 07:16:31.709949 19961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:56455
I20260924 07:16:31.712221 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:16:31.720897 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.714111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.15.127.190:34477)
0924 07:16:31.714244 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.714247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.714375 (+ 128us) server_negotiation.cc:407] Connection header received
0924 07:16:31.714481 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.714484 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.714517 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.714583 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:31.715220 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.715789 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.716663 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.716868 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.716946 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.718341 (+ 1395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:31.718344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:31.718346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:31.718365 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:31.719543 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.719913 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.719916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.719917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.719989 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:31.720222 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:31.720224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:31.720225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:31.720425 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:31.720528 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.720691 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.720780 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:16:31.723608 19961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:31.726784 19961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:31.742098 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.735756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34749 (local address 127.0.0.1:51680)
0924 07:16:31.736542 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.736557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.736662 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.737523 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.737527 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.737589 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:31.737692 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.737701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.739062 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.739065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.740639 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.740644 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.741735 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.741738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.741830 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.741849 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.741909 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.741958 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":263,"thread_start_us":84,"threads_started":1}
I20260924 07:16:31.742209 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.736013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51680 (local address 127.15.127.129:34749)
0924 07:16:31.736943 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.736949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.736968 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:16:31.737054 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.737058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.737211 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.737337 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:31.737827 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.738953 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.740777 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.741617 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.741741 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.741857 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.741945 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.742037 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":108,"threads_started":1}
I20260924 07:16:31.742432 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.737315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.15.127.130:37747)
0924 07:16:31.737694 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.737706 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.737730 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:16:31.737806 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.737811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.738019 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.738262 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:31.738537 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.739425 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.740625 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.741384 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.741517 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.741659 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.742168 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.742287 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":111,"threads_started":1}
I20260924 07:16:31.742331 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.735756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:40513 (local address 127.0.0.1:42780)
0924 07:16:31.736469 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.736519 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.736662 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.737360 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.737365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.737390 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:31.737542 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.737549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.738958 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.738964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.740633 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.740644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.741924 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.741927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.742014 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.742036 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.742108 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.742189 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":193,"thread_start_us":78,"threads_started":1}
I20260924 07:16:31.742097 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.736791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:37747 (local address 127.0.0.1:40386)
0924 07:16:31.737062 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.737077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.737166 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.738246 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.738249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.738312 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:31.738410 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.738417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.739553 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.739558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.740488 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.740498 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.741513 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.741517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.741642 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.741687 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.741797 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.741928 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
I20260924 07:16:31.742738 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.735852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.15.127.131:40513)
0924 07:16:31.736810 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.736823 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.736848 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:16:31.736924 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.736930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.737167 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.737337 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:31.737692 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.738835 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.740777 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.741617 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.741741 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.741857 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.742419 (+ 562us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.742555 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":843,"thread_start_us":161,"threads_started":1}
I20260924 07:16:31.745532 20351 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3251cb56164ee6b42da5bf22019204 (DEFAULT_TABLE table=test-table [id=73bc77846125425fa559eefffc74e4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:31.745596 20218 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3251cb56164ee6b42da5bf22019204 (DEFAULT_TABLE table=test-table [id=73bc77846125425fa559eefffc74e4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:31.745596 20085 tablet_service.cc:1467] Processing CreateTablet for tablet 5b3251cb56164ee6b42da5bf22019204 (DEFAULT_TABLE table=test-table [id=73bc77846125425fa559eefffc74e4b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:31.746560 20351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3251cb56164ee6b42da5bf22019204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:31.746628 20085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3251cb56164ee6b42da5bf22019204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:31.746639 20218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b3251cb56164ee6b42da5bf22019204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:31.752583 20434 tablet_bootstrap.cc:492] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Bootstrap starting.
I20260924 07:16:31.754688 20436 tablet_bootstrap.cc:492] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: Bootstrap starting.
I20260924 07:16:31.755035 20434 tablet_bootstrap.cc:654] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:31.756155 20434 log.cc:824] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:31.756517 20436 tablet_bootstrap.cc:654] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:31.756896 20435 tablet_bootstrap.cc:492] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Bootstrap starting.
I20260924 07:16:31.757884 20436 log.cc:824] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:31.758704 20434 tablet_bootstrap.cc:492] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: No bootstrap required, opened a new log
I20260924 07:16:31.758957 20434 ts_tablet_manager.cc:1397] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:16:31.758991 20435 tablet_bootstrap.cc:654] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:31.760033 20435 log.cc:824] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:31.760473 20436 tablet_bootstrap.cc:492] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: No bootstrap required, opened a new log
I20260924 07:16:31.760661 20436 ts_tablet_manager.cc:1397] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 07:16:31.761758 20435 tablet_bootstrap.cc:492] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: No bootstrap required, opened a new log
I20260924 07:16:31.761948 20435 ts_tablet_manager.cc:1397] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:16:31.762933 20434 raft_consensus.cc:348] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.763229 20434 raft_consensus.cc:374] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:31.763320 20434 raft_consensus.cc:729] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4937b3b6e2e6474c9e4af0a57f405ac0, State: Initialized, Role: FOLLOWER
I20260924 07:16:31.763813 20434 consensus_queue.cc:260] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.764508 20436 raft_consensus.cc:348] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.764808 20436 raft_consensus.cc:374] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:31.764878 20434 ts_tablet_manager.cc:1428] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260924 07:16:31.764899 20436 raft_consensus.cc:729] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f274715d511743c3b765da7883786fd0, State: Initialized, Role: FOLLOWER
I20260924 07:16:31.765394 20417 heartbeater.cc:503] Master 127.15.127.190:34477 was elected leader, sending a full tablet report...
I20260924 07:16:31.765462 20436 consensus_queue.cc:260] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.766161 20151 heartbeater.cc:503] Master 127.15.127.190:34477 was elected leader, sending a full tablet report...
I20260924 07:16:31.766506 20436 ts_tablet_manager.cc:1428] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:16:31.765241 20435 raft_consensus.cc:348] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.766742 20435 raft_consensus.cc:374] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:31.766819 20435 raft_consensus.cc:729] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc707ccccba34689870bd68d4df7ad0f, State: Initialized, Role: FOLLOWER
I20260924 07:16:31.767269 20435 consensus_queue.cc:260] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.767930 20284 heartbeater.cc:503] Master 127.15.127.190:34477 was elected leader, sending a full tablet report...
I20260924 07:16:31.768229 20435 ts_tablet_manager.cc:1428] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260924 07:16:31.769172 20152 tablet.cc:2369] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:31.781512 20442 raft_consensus.cc:482] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:31.781697 20442 raft_consensus.cc:504] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.782764 20442 leader_election.cc:288] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4937b3b6e2e6474c9e4af0a57f405ac0 (127.15.127.131:40513), f274715d511743c3b765da7883786fd0 (127.15.127.129:34749)
I20260924 07:16:31.786608 20286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.782988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:40513 (local address 127.15.127.130:60909)
0924 07:16:31.783109 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.783125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.783233 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.783495 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.783498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.783514 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:31.783639 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.783652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.784496 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.784499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.785435 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.785445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.786185 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.786189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.786295 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.786321 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.786405 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.786474 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":1}
I20260924 07:16:31.786652 20430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.783108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:60909 (local address 127.15.127.131:40513)
0924 07:16:31.783275 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.783281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.783297 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:31.783356 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.783358 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.783394 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.783483 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:31.783761 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.784392 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.785555 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.786063 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.786153 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.786193 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.786436 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.786503 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 07:16:31.786856 20443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.782988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34749 (local address 127.15.127.130:54449)
0924 07:16:31.783520 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.783531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:31.783627 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:31.783894 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:31.783897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:31.783913 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:31.784016 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.784022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.784813 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.784816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.785750 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:31.785760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.786448 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.786452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:31.786542 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.786564 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:16:31.786642 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:16:31.786713 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":211,"thread_start_us":153,"threads_started":1}
I20260924 07:16:31.786895 20431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.783112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:54449 (local address 127.15.127.129:34749)
0924 07:16:31.783265 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.783269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.783597 (+ 328us) server_negotiation.cc:407] Connection header received
0924 07:16:31.783728 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.783733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.783788 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.783852 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:31.784131 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.784703 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.785873 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.786331 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.786438 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.786481 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.786684 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.786758 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:16:31.787390 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3251cb56164ee6b42da5bf22019204" candidate_uuid: "dc707ccccba34689870bd68d4df7ad0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" is_pre_election: true
I20260924 07:16:31.787539 20105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3251cb56164ee6b42da5bf22019204" candidate_uuid: "dc707ccccba34689870bd68d4df7ad0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f274715d511743c3b765da7883786fd0" is_pre_election: true
I20260924 07:16:31.787727 20371 raft_consensus.cc:2454] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc707ccccba34689870bd68d4df7ad0f in term 0.
I20260924 07:16:31.787849 20105 raft_consensus.cc:2454] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc707ccccba34689870bd68d4df7ad0f in term 0.
I20260924 07:16:31.788276 20172 leader_election.cc:302] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4937b3b6e2e6474c9e4af0a57f405ac0, dc707ccccba34689870bd68d4df7ad0f, f274715d511743c3b765da7883786fd0; no voters:
I20260924 07:16:31.788594 20442 raft_consensus.cc:2781] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:31.788699 20442 raft_consensus.cc:482] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:31.788744 20442 raft_consensus.cc:3037] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:31.789803 20442 raft_consensus.cc:504] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.790155 20442 leader_election.cc:288] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [CANDIDATE]: Term 1 election: Requested vote from peers 4937b3b6e2e6474c9e4af0a57f405ac0 (127.15.127.131:40513), f274715d511743c3b765da7883786fd0 (127.15.127.129:34749)
I20260924 07:16:31.790519 20105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3251cb56164ee6b42da5bf22019204" candidate_uuid: "dc707ccccba34689870bd68d4df7ad0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f274715d511743c3b765da7883786fd0"
I20260924 07:16:31.790501 20371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b3251cb56164ee6b42da5bf22019204" candidate_uuid: "dc707ccccba34689870bd68d4df7ad0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0"
I20260924 07:16:31.790652 20105 raft_consensus.cc:3037] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:31.790655 20371 raft_consensus.cc:3037] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:31.791743 20105 raft_consensus.cc:2454] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc707ccccba34689870bd68d4df7ad0f in term 1.
I20260924 07:16:31.791800 20371 raft_consensus.cc:2454] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc707ccccba34689870bd68d4df7ad0f in term 1.
I20260924 07:16:31.792083 20171 leader_election.cc:302] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc707ccccba34689870bd68d4df7ad0f, f274715d511743c3b765da7883786fd0; no voters:
I20260924 07:16:31.792272 20442 raft_consensus.cc:2781] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:31.792512 20442 raft_consensus.cc:686] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 1 LEADER]: Becoming Leader. State: Replica: dc707ccccba34689870bd68d4df7ad0f, State: Running, Role: LEADER
I20260924 07:16:31.792869 20442 consensus_queue.cc:237] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } }
I20260924 07:16:31.795979 19958 catalog_manager.cc:5495] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f reported cstate change: term changed from 0 to 1, leader changed from <none> to dc707ccccba34689870bd68d4df7ad0f (127.15.127.130). New cstate: current_term: 1 leader_uuid: "dc707ccccba34689870bd68d4df7ad0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc707ccccba34689870bd68d4df7ad0f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 37747 } health_report { overall_health: HEALTHY } } }
I20260924 07:16:31.811635 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.808337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40388 (local address 127.15.127.130:37747)
0924 07:16:31.808459 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:31.808463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:31.808475 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:31.808546 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:31.808548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:31.808600 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:31.808701 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:31.808928 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.809473 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.810301 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:31.810513 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:31.810596 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:31.810864 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:31.811302 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:31.811395 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:31.811444 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:16:31.811494 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:16:31.817891 20371 raft_consensus.cc:1261] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 1 FOLLOWER]: Refusing update from remote peer dc707ccccba34689870bd68d4df7ad0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:31.817891 20105 raft_consensus.cc:1261] T 5b3251cb56164ee6b42da5bf22019204 P f274715d511743c3b765da7883786fd0 [term 1 FOLLOWER]: Refusing update from remote peer dc707ccccba34689870bd68d4df7ad0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:31.818601 20442 consensus_queue.cc:1035] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f274715d511743c3b765da7883786fd0" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:31.818867 20444 consensus_queue.cc:1035] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4937b3b6e2e6474c9e4af0a57f405ac0" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:31.827071 20449 mvcc.cc:205] Tried to move back new op lower bound from 7332799249678405632 to 7332799249586348032. Current Snapshot: MvccSnapshot[applied={T|T < 7332799249678405632}]
I20260924 07:16:31.829622 20448 mvcc.cc:205] Tried to move back new op lower bound from 7332799249678405632 to 7332799249586348032. Current Snapshot: MvccSnapshot[applied={T|T < 7332799249678405632 or (T in {7332799249678405632})}]
I20260924 07:16:31.836342 19961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:31.836537 19961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:31.839257 19961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7828d9a3d12b485ab18bb48f1618751f: Sending DeleteTablet for 3 replicas of tablet 5b3251cb56164ee6b42da5bf22019204
I20260924 07:16:31.840137 20218 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3251cb56164ee6b42da5bf22019204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:31 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40386
I20260924 07:16:31.840147 20085 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3251cb56164ee6b42da5bf22019204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:31 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:51680
I20260924 07:16:31.840670 20351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b3251cb56164ee6b42da5bf22019204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:31 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42780
I20260924 07:16:31.840917 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20023
I20260924 07:16:31.841498 20456 tablet_replica.cc:330] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: stopping tablet replica
I20260924 07:16:31.841698 20458 tablet_replica.cc:330] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: stopping tablet replica
I20260924 07:16:31.841917 20456 raft_consensus.cc:2229] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:31.842121 20458 raft_consensus.cc:2229] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:31.842446 20456 raft_consensus.cc:2258] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:31.842528 20458 raft_consensus.cc:2258] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:31.843997 20458 ts_tablet_manager.cc:1905] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:31.843999 20456 ts_tablet_manager.cc:1905] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:31.846710 20456 ts_tablet_manager.cc:1918] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:31.846837 20456 log.cc:1196] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/5b3251cb56164ee6b42da5bf22019204
I20260924 07:16:31.846876 20458 ts_tablet_manager.cc:1918] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:31.847015 20458 log.cc:1196] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/5b3251cb56164ee6b42da5bf22019204
I20260924 07:16:31.847232 20456 ts_tablet_manager.cc:1939] T 5b3251cb56164ee6b42da5bf22019204 P dc707ccccba34689870bd68d4df7ad0f: Deleting consensus metadata
I20260924 07:16:31.847391 20458 ts_tablet_manager.cc:1939] T 5b3251cb56164ee6b42da5bf22019204 P 4937b3b6e2e6474c9e4af0a57f405ac0: Deleting consensus metadata
I20260924 07:16:31.848430 19946 catalog_manager.cc:4841] TS dc707ccccba34689870bd68d4df7ad0f (127.15.127.130:37747): tablet 5b3251cb56164ee6b42da5bf22019204 (table test-table [id=73bc77846125425fa559eefffc74e4b5]) successfully deleted
I20260924 07:16:31.848698 19946 catalog_manager.cc:4841] TS 4937b3b6e2e6474c9e4af0a57f405ac0 (127.15.127.131:40513): tablet 5b3251cb56164ee6b42da5bf22019204 (table test-table [id=73bc77846125425fa559eefffc74e4b5]) successfully deleted
I20260924 07:16:31.853281 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20156
W20260924 07:16:31.853274 19945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:34749 (error 108)
I20260924 07:16:31.855599 20429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:31.855214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34749 (local address 127.0.0.1:51684)
0924 07:16:31.855394 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:31.855485 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 07:16:31.855808 19945 catalog_manager.cc:4584] TS f274715d511743c3b765da7883786fd0 (127.15.127.129:34749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5b3251cb56164ee6b42da5bf22019204: Network error: Client connection negotiation failed: client connection to 127.15.127.129:34749: connect: Connection refused (error 111)
I20260924 07:16:31.859361 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20289
I20260924 07:16:31.864889 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 19930
2026-09-24T07:16:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2233 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T07:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:31Z Disabled control of system clock
I20260924 07:16:31.897447 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:40273
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:16:32.009158 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:32.009442 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:32.009480 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:32.013561 20463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:32.013620 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:32.013661 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:32.013679 20463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:32.013695 20463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:32.020882 20463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:40273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:32.022413 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:32.023797 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:32.032250 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:32.032425 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.032574 20463 server_base.cc:1034] running on GCE node
W20260924 07:16:32.032440 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.033236 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:32.034359 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:32.035545 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790234192035513 us; error 46 us; skew 500 ppm
I20260924 07:16:32.042081 20463 webserver.cc:460] Webserver started at http://127.15.127.190:39329/ using document root <none> and password file <none>
I20260924 07:16:32.042795 20463 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:32.042891 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:32.043128 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:32.045320 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "82398e4542b54fd4981f85170787b654"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "df712cc3da06ded9c2688d56885e1519"
server_key_iv: "c3bc8ed509936e70323967b3f2adbfba"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.045894 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "82398e4542b54fd4981f85170787b654"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "df712cc3da06ded9c2688d56885e1519"
server_key_iv: "c3bc8ed509936e70323967b3f2adbfba"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.049597 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:16:32.052065 20477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.053494 20463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:32.053648 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "82398e4542b54fd4981f85170787b654"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "df712cc3da06ded9c2688d56885e1519"
server_key_iv: "c3bc8ed509936e70323967b3f2adbfba"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.053766 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:32.058341 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:32.058946 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:32.059130 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:32.066056 20463 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:40273
I20260924 07:16:32.066072 20529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:40273 every 8 connection(s)
I20260924 07:16:32.066992 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:32.070195 20530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:32.073261 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20463
I20260924 07:16:32.073403 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:32.073678 15870 external_mini_cluster.cc:1428] Setting key f55b06e9f02cf4f3e842a77ca2743f33
I20260924 07:16:32.076347 20530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654: Bootstrap starting.
I20260924 07:16:32.078902 20530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:32.079442 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.075117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.15.127.190:40273)
0924 07:16:32.075820 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.075836 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.075871 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:16:32.076540 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.076559 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.076927 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.077339 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.078095 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.078100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.078111 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.078121 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.078175 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.078191 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.078309 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.078441 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.078691 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":148,"threads_started":1}
I20260924 07:16:32.079720 20530 log.cc:824] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:32.081771 20530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654: No bootstrap required, opened a new log
I20260924 07:16:32.085170 20530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82398e4542b54fd4981f85170787b654" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40273 } }
I20260924 07:16:32.085391 20530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:32.085424 20530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82398e4542b54fd4981f85170787b654, State: Initialized, Role: FOLLOWER
I20260924 07:16:32.085837 20530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82398e4542b54fd4981f85170787b654" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40273 } }
I20260924 07:16:32.085958 20530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:32.086002 20530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:32.086072 20530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:32.086918 20530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82398e4542b54fd4981f85170787b654" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40273 } }
I20260924 07:16:32.087306 20530 leader_election.cc:302] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82398e4542b54fd4981f85170787b654; no voters:
I20260924 07:16:32.087680 20530 leader_election.cc:288] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:32.087742 20535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:32.087944 20535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [term 1 LEADER]: Becoming Leader. State: Replica: 82398e4542b54fd4981f85170787b654, State: Running, Role: LEADER
I20260924 07:16:32.088269 20535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82398e4542b54fd4981f85170787b654" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40273 } }
I20260924 07:16:32.088986 20530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:32.090019 20536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82398e4542b54fd4981f85170787b654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82398e4542b54fd4981f85170787b654" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40273 } } }
I20260924 07:16:32.090051 20537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82398e4542b54fd4981f85170787b654. Latest consensus state: current_term: 1 leader_uuid: "82398e4542b54fd4981f85170787b654" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82398e4542b54fd4981f85170787b654" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40273 } } }
I20260924 07:16:32.090142 20536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:32.090166 20537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:32.091120 20544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:32.093897 20544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:32.098825 20544 catalog_manager.cc:1347] Generated new cluster ID: 2dea0b13881c408b95afa85a91df6641
I20260924 07:16:32.098891 20544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:32.131950 20544 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:32.132309 20544 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:32.151604 20544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654: Generated new TSK 0
I20260924 07:16:32.152415 20544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:16:32.160398 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:40273
--builtin_ntp_servers=127.15.127.148:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:16:32.280488 20554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:32.280786 20554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:32.280826 20554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:32.285128 20554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:32.285202 20554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:32.285290 20554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:32.290768 20554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:32.292008 20554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:32.293578 20554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:32.303478 20559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.303535 20554 server_base.cc:1034] running on GCE node
W20260924 07:16:32.303443 20560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:32.303443 20562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.304124 20554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:32.304792 20554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:32.305985 20554 hybrid_clock.cc:648] HybridClock initialized: now 1790234192305972 us; error 65 us; skew 500 ppm
I20260924 07:16:32.307938 20554 webserver.cc:460] Webserver started at http://127.15.127.129:44181/ using document root <none> and password file <none>
I20260924 07:16:32.308533 20554 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:32.308629 20554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:32.308866 20554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:32.311017 20554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "29c929012974437291e7e29bfb5c72a2"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "2399cce3110966ba848da524f0d8480a"
server_key_iv: "52933ab81eacd028e1e64a3f6419eea4"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.311605 20554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "29c929012974437291e7e29bfb5c72a2"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "2399cce3110966ba848da524f0d8480a"
server_key_iv: "52933ab81eacd028e1e64a3f6419eea4"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.315387 20554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 07:16:32.318087 20568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.319344 20554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:32.319491 20554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "29c929012974437291e7e29bfb5c72a2"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "2399cce3110966ba848da524f0d8480a"
server_key_iv: "52933ab81eacd028e1e64a3f6419eea4"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.319617 20554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:32.324460 20554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:32.325205 20554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:32.325394 20554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:32.325973 20554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:32.326959 20554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:32.327030 20554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.327095 20554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:32.327149 20554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.336011 20554 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:44687
I20260924 07:16:32.336030 20681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:44687 every 8 connection(s)
I20260924 07:16:32.336969 20554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:32.337283 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20554
I20260924 07:16:32.337430 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:32.337770 15870 external_mini_cluster.cc:1428] Setting key 09b3e6c93b234c90aea78f0edaf26220
I20260924 07:16:32.339985 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:40273
--builtin_ntp_servers=127.15.127.148:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 07:16:32.343966 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.339009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:55597 (local address 127.15.127.190:40273)
0924 07:16:32.339152 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.339159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.340532 (+ 1373us) server_negotiation.cc:407] Connection header received
0924 07:16:32.342025 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.342028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.342073 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.342168 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.343358 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.343362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.343364 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.343384 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.343405 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.343412 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.343509 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.343708 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.343764 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:16:32.344933 20684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.339612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:40273 (local address 127.15.127.129:55597)
0924 07:16:32.340377 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.340431 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.341799 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.342360 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.342367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.342461 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.343094 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.343096 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.343140 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.343210 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.343500 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.343509 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.343517 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.343738 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.343943 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":152,"threads_started":1}
I20260924 07:16:32.346385 20682 heartbeater.cc:347] Connected to a master server at 127.15.127.190:40273
I20260924 07:16:32.346652 20682 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:32.347095 20682 heartbeater.cc:511] Master 127.15.127.190:40273 requested a full tablet report, sending...
I20260924 07:16:32.348687 20494 ts_manager.cc:194] Registered new tserver with Master: 29c929012974437291e7e29bfb5c72a2 (127.15.127.129:44687)
W20260924 07:16:32.452209 20685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:32.452515 20685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:32.452589 20685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:32.456640 20685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:32.456727 20685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:32.456815 20685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:32.461727 20685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:32.462836 20685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:32.464250 20685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:32.474263 20693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:32.474284 20690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.474351 20685 server_base.cc:1034] running on GCE node
W20260924 07:16:32.474246 20691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.475006 20685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:32.475692 20685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:32.476861 20685 hybrid_clock.cc:648] HybridClock initialized: now 1790234192476842 us; error 34 us; skew 500 ppm
I20260924 07:16:32.478734 20685 webserver.cc:460] Webserver started at http://127.15.127.130:43509/ using document root <none> and password file <none>
I20260924 07:16:32.479342 20685 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:32.479405 20685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:32.479673 20685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:32.481891 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "9b9dc70115434389aff6d4a596bfc6db"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "419e116b5ec9d0ca20f968c5ddfceb3d"
server_key_iv: "2d5728adff3cb9bea24719d7f3687563"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.482473 20685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "9b9dc70115434389aff6d4a596bfc6db"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "419e116b5ec9d0ca20f968c5ddfceb3d"
server_key_iv: "2d5728adff3cb9bea24719d7f3687563"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.486372 20685 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:16:32.489073 20699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.490495 20685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:32.490623 20685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "9b9dc70115434389aff6d4a596bfc6db"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "419e116b5ec9d0ca20f968c5ddfceb3d"
server_key_iv: "2d5728adff3cb9bea24719d7f3687563"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.490780 20685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:32.495464 20685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:32.496044 20685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:32.496186 20685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:32.496770 20685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:32.497717 20685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:32.497766 20685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.497802 20685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:32.497824 20685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.506649 20685 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:43041
I20260924 07:16:32.506680 20812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:43041 every 8 connection(s)
I20260924 07:16:32.507598 20685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:32.513774 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.509423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:52133 (local address 127.15.127.190:40273)
0924 07:16:32.509558 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.509562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.510626 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 07:16:32.511988 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.511991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.512034 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.512113 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.513293 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.513296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.513299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.513311 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.513333 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.513344 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.513443 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.513591 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.513636 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:16:32.514605 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.509847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:40273 (local address 127.15.127.130:52133)
0924 07:16:32.510477 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.510517 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.511771 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.512271 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.512277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.512364 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.512978 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.512979 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.513074 (+ 95us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.513145 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.513436 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.513444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.513452 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.513661 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.513894 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":106,"threads_started":1}
I20260924 07:16:32.515707 20813 heartbeater.cc:347] Connected to a master server at 127.15.127.190:40273
I20260924 07:16:32.515889 20813 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:32.516229 20813 heartbeater.cc:511] Master 127.15.127.190:40273 requested a full tablet report, sending...
I20260924 07:16:32.517220 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20685
I20260924 07:16:32.517234 20494 ts_manager.cc:194] Registered new tserver with Master: 9b9dc70115434389aff6d4a596bfc6db (127.15.127.130:43041)
I20260924 07:16:32.517333 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:32.517664 15870 external_mini_cluster.cc:1428] Setting key 6bb43b4174e3fae00ad342eff7d6c117
I20260924 07:16:32.519460 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:40273
--builtin_ntp_servers=127.15.127.148:42813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:16:32.632910 20816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:32.633284 20816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:32.633328 20816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:32.637462 20816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:32.637528 20816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:32.637588 20816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:32.642504 20816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:32.643703 20816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:32.645543 20816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:32.655224 20821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:32.655187 20822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.655272 20816 server_base.cc:1034] running on GCE node
W20260924 07:16:32.655203 20824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:32.655962 20816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:32.656658 20816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:32.657879 20816 hybrid_clock.cc:648] HybridClock initialized: now 1790234192657847 us; error 46 us; skew 500 ppm
I20260924 07:16:32.659816 20816 webserver.cc:460] Webserver started at http://127.15.127.131:44723/ using document root <none> and password file <none>
I20260924 07:16:32.660356 20816 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:32.660418 20816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:32.660580 20816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:32.662669 20816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "e27e240256774aa2b06cc9f38b0a3db9"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "f34c1689d560e65f0959205b7c3bbc7a"
server_key_iv: "57a576d0e34ce8ae9baf1837dc3304cb"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.663192 20816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "e27e240256774aa2b06cc9f38b0a3db9"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "f34c1689d560e65f0959205b7c3bbc7a"
server_key_iv: "57a576d0e34ce8ae9baf1837dc3304cb"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.666893 20816 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 07:16:32.669305 20830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.670691 20816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:16:32.670821 20816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "e27e240256774aa2b06cc9f38b0a3db9"
format_stamp: "Formatted at 2026-09-24 07:16:32 on dist-test-slave-zx46"
server_key: "f34c1689d560e65f0959205b7c3bbc7a"
server_key_iv: "57a576d0e34ce8ae9baf1837dc3304cb"
server_key_version: "encryptionkey@0"
I20260924 07:16:32.670936 20816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:32.675462 20816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:32.676046 20816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:32.676229 20816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:32.676752 20816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:32.677717 20816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:32.677788 20816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.677855 20816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:32.677907 20816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:32.686626 20816 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:37807
I20260924 07:16:32.686646 20943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:37807 every 8 connection(s)
I20260924 07:16:32.687544 20816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:32.693990 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.689421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:59875 (local address 127.15.127.190:40273)
0924 07:16:32.689575 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.689584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.690761 (+ 1177us) server_negotiation.cc:407] Connection header received
0924 07:16:32.692242 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.692246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.692290 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.692372 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.693495 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.693498 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.693501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.693513 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.693536 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.693547 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.693663 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.693842 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.693885 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:16:32.694912 20946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.689856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:40273 (local address 127.15.127.131:59875)
0924 07:16:32.690616 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.690658 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.692020 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.692539 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.692547 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.692639 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.693244 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.693246 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.693284 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.693350 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.693639 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.693647 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.693653 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.693857 (+ 204us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.694065 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":133,"threads_started":1}
I20260924 07:16:32.695729 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20816
I20260924 07:16:32.695837 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:32.696132 15870 external_mini_cluster.cc:1428] Setting key d9663ca3ff4acc7523730a7156119650
I20260924 07:16:32.696111 20944 heartbeater.cc:347] Connected to a master server at 127.15.127.190:40273
I20260924 07:16:32.696344 20944 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:32.696770 20944 heartbeater.cc:511] Master 127.15.127.190:40273 requested a full tablet report, sending...
I20260924 07:16:32.697899 20493 ts_manager.cc:194] Registered new tserver with Master: e27e240256774aa2b06cc9f38b0a3db9 (127.15.127.131:37807)
I20260924 07:16:32.710022 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:16:32.713560 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.711947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.15.127.190:40273)
0924 07:16:32.712090 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.712100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.712297 (+ 197us) server_negotiation.cc:407] Connection header received
0924 07:16:32.712396 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.712398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.712432 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.712500 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.713243 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.713246 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.713248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.713259 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.713278 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.713285 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.713356 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.713425 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.713458 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:16:32.716514 20493 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:32.719601 20493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:32.731329 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.728259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:44687 (local address 127.0.0.1:45750)
0924 07:16:32.728899 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.728952 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.729160 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.729708 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.729715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.729796 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.730371 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.730374 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.730394 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.730470 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.730949 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.730958 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.730963 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.731081 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.731153 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":86,"threads_started":1}
I20260924 07:16:32.731807 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.728382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.15.127.129:44687)
0924 07:16:32.728722 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.728734 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.729071 (+ 337us) server_negotiation.cc:407] Connection header received
0924 07:16:32.729305 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.729312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.729508 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.730100 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.730786 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.730790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.730799 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.730806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.730839 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.730855 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.730977 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.731381 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.731662 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":114,"threads_started":1}
I20260924 07:16:32.732841 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.728330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:37807 (local address 127.0.0.1:52098)
0924 07:16:32.729884 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.729898 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.729984 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.730969 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.730971 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.730984 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.732146 (+ 1162us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.732149 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.732162 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.732188 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.732539 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.732541 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.732544 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.732639 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.732678 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"thread_start_us":100,"threads_started":1}
I20260924 07:16:32.732862 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.729100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.15.127.131:37807)
0924 07:16:32.729753 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.729767 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.729971 (+ 204us) server_negotiation.cc:407] Connection header received
0924 07:16:32.730113 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.730120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.730315 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.730420 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.732345 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.732351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.732363 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.732375 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.732417 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.732448 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.732556 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.732638 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.732717 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":154,"threads_started":1}
I20260924 07:16:32.733351 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.729488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.15.127.130:43041)
0924 07:16:32.729897 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.729909 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.731215 (+ 1306us) server_negotiation.cc:407] Connection header received
0924 07:16:32.731282 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.731288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.731473 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.731616 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.732434 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.732440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.732452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.732464 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.732528 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.732565 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.732664 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.732906 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.733230 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":113,"threads_started":1}
I20260924 07:16:32.733582 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.729071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:43041 (local address 127.0.0.1:60754)
0924 07:16:32.731111 (+ 2040us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.731134 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.731203 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.731812 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.731814 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.731826 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.732280 (+ 454us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.732283 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.732293 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.732316 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.732717 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.732720 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.732722 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.732789 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.732817 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1977,"thread_start_us":130,"threads_started":1}
I20260924 07:16:32.735065 20616 tablet_service.cc:1467] Processing CreateTablet for tablet dcd213d5b91444a099c77b361898fcc5 (DEFAULT_TABLE table=test-table [id=fc96dd7c8b564b7886fad15cde298d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:32.735466 20878 tablet_service.cc:1467] Processing CreateTablet for tablet dcd213d5b91444a099c77b361898fcc5 (DEFAULT_TABLE table=test-table [id=fc96dd7c8b564b7886fad15cde298d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:32.736168 20616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd213d5b91444a099c77b361898fcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:32.736443 20878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd213d5b91444a099c77b361898fcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:32.738336 20747 tablet_service.cc:1467] Processing CreateTablet for tablet dcd213d5b91444a099c77b361898fcc5 (DEFAULT_TABLE table=test-table [id=fc96dd7c8b564b7886fad15cde298d1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:32.739355 20747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd213d5b91444a099c77b361898fcc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:32.742674 20961 tablet_bootstrap.cc:492] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: Bootstrap starting.
I20260924 07:16:32.744207 20962 tablet_bootstrap.cc:492] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Bootstrap starting.
I20260924 07:16:32.744850 20961 tablet_bootstrap.cc:654] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:32.745723 20961 log.cc:824] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:32.746368 20962 tablet_bootstrap.cc:654] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:32.747042 20963 tablet_bootstrap.cc:492] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: Bootstrap starting.
I20260924 07:16:32.747189 20962 log.cc:824] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:32.747804 20961 tablet_bootstrap.cc:492] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: No bootstrap required, opened a new log
I20260924 07:16:32.748009 20961 ts_tablet_manager.cc:1397] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:16:32.749099 20963 tablet_bootstrap.cc:654] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:32.749212 20962 tablet_bootstrap.cc:492] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: No bootstrap required, opened a new log
I20260924 07:16:32.749390 20962 ts_tablet_manager.cc:1397] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:16:32.749893 20963 log.cc:824] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:32.751674 20961 raft_consensus.cc:348] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.751943 20963 tablet_bootstrap.cc:492] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: No bootstrap required, opened a new log
I20260924 07:16:32.751948 20961 raft_consensus.cc:374] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:32.752017 20961 raft_consensus.cc:729] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c929012974437291e7e29bfb5c72a2, State: Initialized, Role: FOLLOWER
I20260924 07:16:32.752137 20963 ts_tablet_manager.cc:1397] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260924 07:16:32.752465 20961 consensus_queue.cc:260] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.753126 20682 heartbeater.cc:503] Master 127.15.127.190:40273 was elected leader, sending a full tablet report...
I20260924 07:16:32.753787 20961 ts_tablet_manager.cc:1428] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:16:32.753661 20962 raft_consensus.cc:348] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.753912 20962 raft_consensus.cc:374] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:32.753959 20962 raft_consensus.cc:729] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27e240256774aa2b06cc9f38b0a3db9, State: Initialized, Role: FOLLOWER
I20260924 07:16:32.754441 20962 consensus_queue.cc:260] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.755096 20944 heartbeater.cc:503] Master 127.15.127.190:40273 was elected leader, sending a full tablet report...
I20260924 07:16:32.755525 20962 ts_tablet_manager.cc:1428] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:16:32.756484 20963 raft_consensus.cc:348] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.756742 20963 raft_consensus.cc:374] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:32.756853 20963 raft_consensus.cc:729] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9dc70115434389aff6d4a596bfc6db, State: Initialized, Role: FOLLOWER
I20260924 07:16:32.757413 20963 consensus_queue.cc:260] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.758160 20813 heartbeater.cc:503] Master 127.15.127.190:40273 was elected leader, sending a full tablet report...
I20260924 07:16:32.758432 20963 ts_tablet_manager.cc:1428] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260924 07:16:32.758504 20814 tablet.cc:2369] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:32.838500 20683 tablet.cc:2369] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:32.938539 20945 tablet.cc:2369] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:32.995115 20969 raft_consensus.cc:482] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:32.995349 20969 raft_consensus.cc:504] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:32.996523 20969 leader_election.cc:288] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29c929012974437291e7e29bfb5c72a2 (127.15.127.129:44687), e27e240256774aa2b06cc9f38b0a3db9 (127.15.127.131:37807)
I20260924 07:16:32.998574 20957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.997073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:38341 (local address 127.15.127.129:44687)
0924 07:16:32.997230 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.997234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.997247 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:32.997290 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.997292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.997323 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.997390 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.998127 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.998131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.998139 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.998150 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.998195 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.998207 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.998303 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.998396 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.998448 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 07:16:32.998598 20959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.996955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:51277 (local address 127.15.127.131:37807)
0924 07:16:32.997125 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:32.997131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:32.997497 (+ 366us) server_negotiation.cc:407] Connection header received
0924 07:16:32.997542 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:32.997545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:32.997579 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:32.997683 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:32.998240 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:32.998246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:32.998249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:32.998265 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:32.998285 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:32.998294 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:32.998380 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:32.998460 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:16:32.998494 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 07:16:32.998571 20815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.996825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:44687 (local address 127.15.127.130:38341)
0924 07:16:32.996934 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.996951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.997064 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.997448 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.997454 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.997467 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.997955 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.997957 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.997966 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.998013 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.998292 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.998294 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.998296 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.998388 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.998431 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:16:32.998593 20970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:32.996825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:37807 (local address 127.15.127.130:51277)
0924 07:16:32.997393 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:32.997408 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:32.997477 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:32.997702 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:32.997704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:32.997715 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:32.998071 (+ 356us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:16:32.998075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:32.998086 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:16:32.998108 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:32.998373 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:32.998376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:32.998378 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:16:32.998447 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:16:32.998471 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":311,"thread_start_us":93,"threads_started":1}
I20260924 07:16:32.999300 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd213d5b91444a099c77b361898fcc5" candidate_uuid: "9b9dc70115434389aff6d4a596bfc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c929012974437291e7e29bfb5c72a2" is_pre_election: true
I20260924 07:16:32.999298 20898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd213d5b91444a099c77b361898fcc5" candidate_uuid: "9b9dc70115434389aff6d4a596bfc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e240256774aa2b06cc9f38b0a3db9" is_pre_election: true
I20260924 07:16:32.999625 20898 raft_consensus.cc:2454] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9dc70115434389aff6d4a596bfc6db in term 0.
I20260924 07:16:32.999635 20636 raft_consensus.cc:2454] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9dc70115434389aff6d4a596bfc6db in term 0.
I20260924 07:16:33.000149 20701 leader_election.cc:302] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29c929012974437291e7e29bfb5c72a2, 9b9dc70115434389aff6d4a596bfc6db, e27e240256774aa2b06cc9f38b0a3db9; no voters:
I20260924 07:16:33.000475 20969 raft_consensus.cc:2781] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:33.000574 20969 raft_consensus.cc:482] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:33.000629 20969 raft_consensus.cc:3037] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:33.001736 20969 raft_consensus.cc:504] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:33.002104 20969 leader_election.cc:288] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [CANDIDATE]: Term 1 election: Requested vote from peers 29c929012974437291e7e29bfb5c72a2 (127.15.127.129:44687), e27e240256774aa2b06cc9f38b0a3db9 (127.15.127.131:37807)
I20260924 07:16:33.002475 20636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd213d5b91444a099c77b361898fcc5" candidate_uuid: "9b9dc70115434389aff6d4a596bfc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c929012974437291e7e29bfb5c72a2"
I20260924 07:16:33.002498 20898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd213d5b91444a099c77b361898fcc5" candidate_uuid: "9b9dc70115434389aff6d4a596bfc6db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27e240256774aa2b06cc9f38b0a3db9"
I20260924 07:16:33.002609 20898 raft_consensus.cc:3037] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:33.002609 20636 raft_consensus.cc:3037] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:33.003901 20636 raft_consensus.cc:2454] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9dc70115434389aff6d4a596bfc6db in term 1.
I20260924 07:16:33.003901 20898 raft_consensus.cc:2454] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9dc70115434389aff6d4a596bfc6db in term 1.
I20260924 07:16:33.004197 20701 leader_election.cc:302] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9dc70115434389aff6d4a596bfc6db, e27e240256774aa2b06cc9f38b0a3db9; no voters:
I20260924 07:16:33.004391 20969 raft_consensus.cc:2781] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:33.004611 20969 raft_consensus.cc:686] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [term 1 LEADER]: Becoming Leader. State: Replica: 9b9dc70115434389aff6d4a596bfc6db, State: Running, Role: LEADER
I20260924 07:16:33.004989 20969 consensus_queue.cc:237] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } }
I20260924 07:16:33.008183 20494 catalog_manager.cc:5495] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b9dc70115434389aff6d4a596bfc6db (127.15.127.130). New cstate: current_term: 1 leader_uuid: "9b9dc70115434389aff6d4a596bfc6db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b9dc70115434389aff6d4a596bfc6db" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43041 } health_report { overall_health: HEALTHY } } }
I20260924 07:16:33.027047 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.025449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.15.127.130:43041)
0924 07:16:33.025583 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:33.025587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:33.025602 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:33.025740 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:33.025743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:33.025805 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:33.025901 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:33.026647 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:33.026650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:33.026652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:33.026662 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:33.026682 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:33.026690 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:33.026790 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:33.026856 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:16:33.026899 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:16:33.033278 20898 raft_consensus.cc:1261] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 1 FOLLOWER]: Refusing update from remote peer 9b9dc70115434389aff6d4a596bfc6db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:33.033277 20636 raft_consensus.cc:1261] T dcd213d5b91444a099c77b361898fcc5 P 29c929012974437291e7e29bfb5c72a2 [term 1 FOLLOWER]: Refusing update from remote peer 9b9dc70115434389aff6d4a596bfc6db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:33.034088 20969 consensus_queue.cc:1035] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [LEADER]: Connected to new peer: Peer: permanent_uuid: "29c929012974437291e7e29bfb5c72a2" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:33.034289 20971 consensus_queue.cc:1035] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27e240256774aa2b06cc9f38b0a3db9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:33.041564 20974 mvcc.cc:205] Tried to move back new op lower bound from 7332799254656741376 to 7332799254551396352. Current Snapshot: MvccSnapshot[applied={T|T < 7332799254656741376}]
I20260924 07:16:33.050752 20494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:33.050904 20494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:33.053356 20494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 82398e4542b54fd4981f85170787b654: Sending DeleteTablet for 3 replicas of tablet dcd213d5b91444a099c77b361898fcc5
I20260924 07:16:33.054158 20616 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd213d5b91444a099c77b361898fcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:33 UTC) from {username='slave'} at 127.0.0.1:45750
I20260924 07:16:33.054216 20878 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd213d5b91444a099c77b361898fcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:33 UTC) from {username='slave'} at 127.0.0.1:52098
I20260924 07:16:33.054695 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20554
I20260924 07:16:33.054901 20747 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd213d5b91444a099c77b361898fcc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:33 UTC) from {username='slave'} at 127.0.0.1:60754
I20260924 07:16:33.055298 20984 tablet_replica.cc:330] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: stopping tablet replica
I20260924 07:16:33.055629 20984 raft_consensus.cc:2229] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:33.055925 20984 raft_consensus.cc:2258] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:33.061709 20985 tablet_replica.cc:330] T dcd213d5b91444a099c77b361898fcc5 P 9b9dc70115434389aff6d4a596bfc6db: stopping tablet replica
W20260924 07:16:33.062186 20481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:44687 (error 108)
I20260924 07:16:33.062347 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20685
I20260924 07:16:33.063846 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.063505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:44687 (local address 127.0.0.1:45764)
0924 07:16:33.063644 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:33.063732 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:44687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 07:16:33.064098 20481 catalog_manager.cc:4584] TS 29c929012974437291e7e29bfb5c72a2 (127.15.127.129:44687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcd213d5b91444a099c77b361898fcc5: Network error: Client connection negotiation failed: client connection to 127.15.127.129:44687: connect: Connection refused (error 111)
I20260924 07:16:33.064508 20984 ts_tablet_manager.cc:1905] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:33.066969 20984 ts_tablet_manager.cc:1918] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:33.067093 20984 log.cc:1196] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/dcd213d5b91444a099c77b361898fcc5
I20260924 07:16:33.067435 20984 ts_tablet_manager.cc:1939] T dcd213d5b91444a099c77b361898fcc5 P e27e240256774aa2b06cc9f38b0a3db9: Deleting consensus metadata
I20260924 07:16:33.068638 20479 catalog_manager.cc:4841] TS e27e240256774aa2b06cc9f38b0a3db9 (127.15.127.131:37807): tablet dcd213d5b91444a099c77b361898fcc5 (table test-table [id=fc96dd7c8b564b7886fad15cde298d1c]) successfully deleted
I20260924 07:16:33.069063 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20816
I20260924 07:16:33.070230 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.069945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:43041 (local address 127.0.0.1:60768)
0924 07:16:33.070123 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:33.070167 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:43041: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 07:16:33.074673 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20463
2026-09-24T07:16:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1207 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 07:16:33.096572 15870 mini_oidc.cc:152] Starting JWKS server
I20260924 07:16:33.098176 15870 webserver.cc:460] Webserver started at https://127.0.0.1:42349/ using document root <none> and password file <none>
I20260924 07:16:33.098316 15870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:16:33.098537 15870 webserver.cc:460] Webserver started at http://127.0.0.1:43501/ using document root <none> and password file <none>
2026-09-24T07:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:33Z Disabled control of system clock
I20260924 07:16:33.107522 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33477
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33477
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42349/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/testchainca.cert with env {}
W20260924 07:16:33.218727 20992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:33.219028 20992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:33.219070 20992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:33.223116 20992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:33.223177 20992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:33.223193 20992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:33.223210 20992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:33.223227 20992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:16:33.223242 20992 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:16:33.223256 20992 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42349/jwks.json
I20260924 07:16:33.228549 20992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42349/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:33.229610 20992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:33.230630 20992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:16:33.239303 20992 server_base.cc:1034] running on GCE node
W20260924 07:16:33.239306 20998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:33.239547 20997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:33.239578 21000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:33.240294 20992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:33.241492 20992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:33.242683 20992 hybrid_clock.cc:648] HybridClock initialized: now 1790234193242669 us; error 27 us; skew 500 ppm
I20260924 07:16:33.244623 20992 webserver.cc:460] Webserver started at http://127.15.127.190:35827/ using document root <none> and password file <none>
I20260924 07:16:33.245293 20992 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:33.245390 20992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:33.245625 20992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:33.247692 20992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "e4a9b180163e434cbf35dcf7c674c584"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "b21a0eccbfb774e7db6a46a3562fd601"
server_key_iv: "c97f920b845d6f7f34c69c92e028bbf5"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.248312 20992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "e4a9b180163e434cbf35dcf7c674c584"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "b21a0eccbfb774e7db6a46a3562fd601"
server_key_iv: "c97f920b845d6f7f34c69c92e028bbf5"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.252117 20992 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:16:33.254765 21006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:33.256219 20992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:33.256404 20992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "e4a9b180163e434cbf35dcf7c674c584"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "b21a0eccbfb774e7db6a46a3562fd601"
server_key_iv: "c97f920b845d6f7f34c69c92e028bbf5"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.256544 20992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:33.298841 20992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:33.299645 20992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:33.299867 20992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:33.307657 20992 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33477
I20260924 07:16:33.307691 21058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33477 every 8 connection(s)
I20260924 07:16:33.308696 20992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:33.312016 21059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:33.313829 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 20992
I20260924 07:16:33.313936 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:33.314278 15870 external_mini_cluster.cc:1428] Setting key 983024e6959d5ecdf1406c897c05fc2b
I20260924 07:16:33.318329 21059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584: Bootstrap starting.
I20260924 07:16:33.320815 21059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:33.321650 21059 log.cc:824] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:33.323866 21059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584: No bootstrap required, opened a new log
I20260924 07:16:33.324501 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.315823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.15.127.190:33477)
0924 07:16:33.316496 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:33.316514 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:33.316553 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:16:33.317473 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:33.317495 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:33.317934 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:33.318237 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:33.318669 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.319693 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.320946 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.321530 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.321664 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.322640 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:33.322646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:33.322660 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:33.323069 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:33.323125 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:33.323145 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:33.323224 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:33.323394 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 07:16:33.323619 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":150,"threads_started":1}
I20260924 07:16:33.327490 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:16:33.327623 21059 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a9b180163e434cbf35dcf7c674c584" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33477 } }
I20260924 07:16:33.327860 21059 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:33.327934 21059 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a9b180163e434cbf35dcf7c674c584, State: Initialized, Role: FOLLOWER
I20260924 07:16:33.328359 21059 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a9b180163e434cbf35dcf7c674c584" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33477 } }
I20260924 07:16:33.328533 21059 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:33.328639 21059 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:33.328763 21059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:33.329869 21059 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a9b180163e434cbf35dcf7c674c584" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33477 } }
I20260924 07:16:33.330212 21059 leader_election.cc:302] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4a9b180163e434cbf35dcf7c674c584; no voters:
I20260924 07:16:33.330510 21059 leader_election.cc:288] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:33.330659 21064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:33.330937 21064 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [term 1 LEADER]: Becoming Leader. State: Replica: e4a9b180163e434cbf35dcf7c674c584, State: Running, Role: LEADER
I20260924 07:16:33.331300 21064 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a9b180163e434cbf35dcf7c674c584" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33477 } }
I20260924 07:16:33.331732 21059 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:33.332978 21065 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4a9b180163e434cbf35dcf7c674c584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a9b180163e434cbf35dcf7c674c584" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33477 } } }
I20260924 07:16:33.332962 21066 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4a9b180163e434cbf35dcf7c674c584. Latest consensus state: current_term: 1 leader_uuid: "e4a9b180163e434cbf35dcf7c674c584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4a9b180163e434cbf35dcf7c674c584" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33477 } } }
I20260924 07:16:33.333118 21066 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:33.333118 21065 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:33.333488 21073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:33.336584 21073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:33.342401 21073 catalog_manager.cc:1347] Generated new cluster ID: e93e0b74ccf444c58628d6159ba9f67f
I20260924 07:16:33.342502 21073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:33.357821 21073 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:33.358870 21073 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:33.371110 21073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4a9b180163e434cbf35dcf7c674c584: Generated new TSK 0
I20260924 07:16:33.371960 21073 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:16:33.412869 21089 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 07:16:33.412992 21089 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:16:33.413597 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.389205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.15.127.190:33477)
0924 07:16:33.389327 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:33.389332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:33.389525 (+ 193us) server_negotiation.cc:407] Connection header received
0924 07:16:33.389758 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:33.389761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:33.389823 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:33.389932 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:16:33.390192 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.390735 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.391648 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.391942 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.392051 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.392403 (+ 352us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:16:33.413092 (+ 20689us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:16:33.413435 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46418: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:16:33.413702 21062 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46418: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 07:16:33.414789 21088 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:33.389103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33477 (local address 127.0.0.1:46418)
0924 07:16:33.389439 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:33.389452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:33.389600 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:33.389941 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:33.389944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:33.389951 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:16:33.390086 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:33.390092 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.390848 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.390852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:33.391497 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:33.391505 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.392068 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.392072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:33.392168 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.392210 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:16:33.413361 (+ 21151us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:16:33.413618 (+ 257us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.127.190:33477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":99,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.127.190:33477: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42349/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 07:16:33.415519 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 20992
2026-09-24T07:16:33Z chronyd exiting
I20260924 07:16:33.452826 15870 test_util.cc:175] -----------------------------------------------
I20260924 07:16:33.452926 15870 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790234163642002-15870-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (357 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 07:16:33.454178 15870 mini_oidc.cc:152] Starting JWKS server
I20260924 07:16:33.455190 15870 webserver.cc:460] Webserver started at https://127.0.0.1:36613/ using document root <none> and password file <none>
I20260924 07:16:33.455328 15870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:16:33.455657 15870 webserver.cc:460] Webserver started at http://127.0.0.1:45335/ using document root <none> and password file <none>
2026-09-24T07:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:33Z Disabled control of system clock
I20260924 07:16:33.465476 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44323
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:41697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:44323
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36613/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/testchainca.cert with env {}
W20260924 07:16:33.591845 21095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:33.592293 21095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:33.592427 21095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:33.597481 21095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:33.597596 21095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:33.597631 21095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:33.597684 21095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:33.597729 21095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:16:33.597779 21095 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:16:33.597819 21095 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36613/jwks.json
I20260924 07:16:33.603911 21095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:44323
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36613/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:33.605201 21095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:33.606266 21095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:33.615146 21101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:33.615130 21103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:33.615149 21100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:33.615192 21095 server_base.cc:1034] running on GCE node
I20260924 07:16:33.615937 21095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:33.617110 21095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:33.618299 21095 hybrid_clock.cc:648] HybridClock initialized: now 1790234193618273 us; error 49 us; skew 500 ppm
I20260924 07:16:33.620178 21095 webserver.cc:460] Webserver started at http://127.15.127.190:44861/ using document root <none> and password file <none>
I20260924 07:16:33.620860 21095 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:33.620955 21095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:33.621212 21095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:33.623315 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "402149e206584f3fb173c95bfd99b680"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "611870a4a8301d4f817b1440411b47c2"
server_key_iv: "3fd6747dbcbe65dd9d4d35aac5ae747a"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.623896 21095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "402149e206584f3fb173c95bfd99b680"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "611870a4a8301d4f817b1440411b47c2"
server_key_iv: "3fd6747dbcbe65dd9d4d35aac5ae747a"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.627759 21095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:33.630304 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:33.631632 21095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:33.631791 21095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "402149e206584f3fb173c95bfd99b680"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "611870a4a8301d4f817b1440411b47c2"
server_key_iv: "3fd6747dbcbe65dd9d4d35aac5ae747a"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.631918 21095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:33.653220 21095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:33.653973 21095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:33.654191 21095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:33.661339 21095 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:44323
I20260924 07:16:33.661347 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:44323 every 8 connection(s)
I20260924 07:16:33.662388 21095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:33.665699 21162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:33.671351 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680: Bootstrap starting.
I20260924 07:16:33.671840 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21095
I20260924 07:16:33.671962 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:33.672251 15870 external_mini_cluster.cc:1428] Setting key 4b325a8e821a3765ab513e6a6b316de8
I20260924 07:16:33.674286 21162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:33.675045 21162 log.cc:824] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:33.677066 21162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680: No bootstrap required, opened a new log
I20260924 07:16:33.680539 21162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402149e206584f3fb173c95bfd99b680" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44323 } }
I20260924 07:16:33.680825 21162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:33.680920 21162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 402149e206584f3fb173c95bfd99b680, State: Initialized, Role: FOLLOWER
I20260924 07:16:33.681452 21162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402149e206584f3fb173c95bfd99b680" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44323 } }
I20260924 07:16:33.681643 21162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:33.681727 21162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:33.681849 21162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:33.682075 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.673627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.15.127.190:44323)
0924 07:16:33.674296 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:33.674310 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:33.674377 (+ 67us) server_negotiation.cc:407] Connection header received
0924 07:16:33.675054 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:33.675070 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:33.675598 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:33.676026 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:33.676477 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.677576 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.678516 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.679021 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.679204 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.680108 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:33.680114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:33.680129 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:33.680538 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:33.680596 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:33.680616 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:33.680740 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:33.680861 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:16:33.681168 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":130,"threads_started":1}
I20260924 07:16:33.683068 21162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402149e206584f3fb173c95bfd99b680" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44323 } }
I20260924 07:16:33.683408 21162 leader_election.cc:302] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 402149e206584f3fb173c95bfd99b680; no voters:
I20260924 07:16:33.683774 21162 leader_election.cc:288] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:33.683848 21167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:33.684044 21167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [term 1 LEADER]: Becoming Leader. State: Replica: 402149e206584f3fb173c95bfd99b680, State: Running, Role: LEADER
I20260924 07:16:33.684340 21167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402149e206584f3fb173c95bfd99b680" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44323 } }
I20260924 07:16:33.685066 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:16:33.685103 21162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:33.685555 21167 sys_catalog.cc:455] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 402149e206584f3fb173c95bfd99b680. Latest consensus state: current_term: 1 leader_uuid: "402149e206584f3fb173c95bfd99b680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402149e206584f3fb173c95bfd99b680" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44323 } } }
I20260924 07:16:33.685724 21167 sys_catalog.cc:458] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:33.686177 21171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:33.687661 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "402149e206584f3fb173c95bfd99b680" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "402149e206584f3fb173c95bfd99b680" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44323 } } }
I20260924 07:16:33.687783 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:33.689996 21171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:33.695739 21171 catalog_manager.cc:1347] Generated new cluster ID: 537f62ecd749430d99d87e3fc7030bd0
I20260924 07:16:33.695924 21171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:33.708110 21171 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:33.709141 21171 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:33.716730 21171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 402149e206584f3fb173c95bfd99b680: Generated new TSK 0
I20260924 07:16:33.717516 21171 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:16:33.750694 21192 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 07:16:33.750797 21192 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:16:33.751576 21165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.728279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47124 (local address 127.15.127.190:44323)
0924 07:16:33.728407 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:33.728411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:33.728595 (+ 184us) server_negotiation.cc:407] Connection header received
0924 07:16:33.728799 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:33.728801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:33.728848 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:33.728965 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:16:33.729239 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.729786 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.730676 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.730911 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.731026 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.731358 (+ 332us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:16:33.751099 (+ 19741us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:16:33.751410 (+ 311us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:16:33.751675 21165 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 07:16:33.751714 21191 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:33.728186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:44323 (local address 127.0.0.1:47124)
0924 07:16:33.728505 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:33.728518 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:33.728649 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:33.728965 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:33.728968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:33.728976 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:16:33.729113 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:33.729126 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.729905 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.729909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:33.730546 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:33.730553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.731034 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.731038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:33.731147 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.731171 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:16:33.751328 (+ 20157us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:16:33.751521 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.127.190:44323: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36613/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":111,"threads_started":1}
I20260924 07:16:33.752346 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21095
2026-09-24T07:16:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (337 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 07:16:33.791714 15870 mini_oidc.cc:152] Starting JWKS server
I20260924 07:16:33.792711 15870 webserver.cc:460] Webserver started at https://127.0.0.1:43781/ using document root <none> and password file <none>
I20260924 07:16:33.792829 15870 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:16:33.793195 15870 webserver.cc:460] Webserver started at http://127.0.0.1:36777/ using document root <none> and password file <none>
2026-09-24T07:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:33Z Disabled control of system clock
I20260924 07:16:33.802171 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36129
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:34165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:36129
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43781/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:16:33.913532 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:33.913827 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:33.913865 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:33.917902 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:33.917961 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:33.917977 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:33.917994 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:33.918009 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:16:33.918025 21198 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:16:33.918045 21198 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43781/jwks.json
I20260924 07:16:33.922784 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:34165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:36129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43781/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:33.923830 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:33.925192 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:33.933624 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:33.933754 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:33.933626 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:33.933990 21198 server_base.cc:1034] running on GCE node
I20260924 07:16:33.934613 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:33.935765 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:33.936959 21198 hybrid_clock.cc:648] HybridClock initialized: now 1790234193936945 us; error 28 us; skew 500 ppm
I20260924 07:16:33.938879 21198 webserver.cc:460] Webserver started at http://127.15.127.190:33479/ using document root <none> and password file <none>
I20260924 07:16:33.939473 21198 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:33.939569 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:33.939805 21198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:33.941982 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "2b5435d05db44d6ea910adbe2dbd5d27"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "88577a12f30ea447fe27fcd01f369c60"
server_key_iv: "a21ae71427983a0e9ebf9e2207440dfa"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.942585 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "2b5435d05db44d6ea910adbe2dbd5d27"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "88577a12f30ea447fe27fcd01f369c60"
server_key_iv: "a21ae71427983a0e9ebf9e2207440dfa"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.946316 21198 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:33.948719 21212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:33.950034 21198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:16:33.950201 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "2b5435d05db44d6ea910adbe2dbd5d27"
format_stamp: "Formatted at 2026-09-24 07:16:33 on dist-test-slave-zx46"
server_key: "88577a12f30ea447fe27fcd01f369c60"
server_key_iv: "a21ae71427983a0e9ebf9e2207440dfa"
server_key_version: "encryptionkey@0"
I20260924 07:16:33.950325 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:33.971382 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:33.972169 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:33.972385 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:33.979514 21198 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:36129
I20260924 07:16:33.979519 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:36129 every 8 connection(s)
I20260924 07:16:33.980530 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:33.983767 21265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:33.988216 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21198
I20260924 07:16:33.988341 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:33.988612 15870 external_mini_cluster.cc:1428] Setting key a27d5038d9248e6dd40dd6fa351cb64a
I20260924 07:16:33.989984 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27: Bootstrap starting.
I20260924 07:16:33.992326 21265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:33.993202 21265 log.cc:824] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:33.995170 21265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27: No bootstrap required, opened a new log
I20260924 07:16:33.998658 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:33.990159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60286 (local address 127.15.127.190:36129)
0924 07:16:33.990834 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:33.990849 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:33.990887 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:16:33.991539 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:33.991557 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:33.991971 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:33.992333 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:33.992856 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.994320 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.995242 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:33.995781 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:33.995965 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:33.996850 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:33.996855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:33.996869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:33.997327 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:33.997383 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:33.997403 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:33.997515 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:33.997663 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:16:33.997921 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":140,"threads_started":1}
I20260924 07:16:33.999059 21265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36129 } }
I20260924 07:16:33.999307 21265 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:33.999379 21265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b5435d05db44d6ea910adbe2dbd5d27, State: Initialized, Role: FOLLOWER
I20260924 07:16:33.999852 21265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36129 } }
I20260924 07:16:34.000015 21265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:34.000093 21265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:34.000214 21265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:34.001315 21265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36129 } }
I20260924 07:16:34.001626 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:16:34.001709 21265 leader_election.cc:302] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b5435d05db44d6ea910adbe2dbd5d27; no voters:
I20260924 07:16:34.002070 21265 leader_election.cc:288] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:34.002205 21270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:34.002404 21270 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [term 1 LEADER]: Becoming Leader. State: Replica: 2b5435d05db44d6ea910adbe2dbd5d27, State: Running, Role: LEADER
I20260924 07:16:34.002698 21270 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36129 } }
I20260924 07:16:34.003377 21265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:34.003957 21270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b5435d05db44d6ea910adbe2dbd5d27. Latest consensus state: current_term: 1 leader_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36129 } } }
I20260924 07:16:34.004073 21270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:34.004520 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b5435d05db44d6ea910adbe2dbd5d27" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36129 } } }
I20260924 07:16:34.004635 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:34.005788 21276 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:34.008405 21276 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:34.013944 21276 catalog_manager.cc:1347] Generated new cluster ID: ffa652bdd58b42d5aae6220959acddee
I20260924 07:16:34.014022 21276 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:34.041314 21276 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:34.042284 21276 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:34.046243 21276 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b5435d05db44d6ea910adbe2dbd5d27: Generated new TSK 0
I20260924 07:16:34.046973 21276 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:16:34.061995 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:34.056891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.15.127.190:36129)
0924 07:16:34.057040 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:34.057047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:34.057203 (+ 156us) server_negotiation.cc:407] Connection header received
0924 07:16:34.057376 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:34.057378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:34.057426 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:34.057509 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:16:34.057757 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.058294 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.059536 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.059745 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.059814 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:34.060149 (+ 335us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:16:34.061612 (+ 1463us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:16:34.061857 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60294: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 07:16:34.062067 21268 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60294: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 07:16:34.063809 21294 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:34.056747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36129 (local address 127.0.0.1:60294)
0924 07:16:34.057103 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:34.057117 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:34.057240 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:34.057533 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:34.057536 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:34.057544 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:16:34.057652 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:34.057659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.058404 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.058408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:34.059411 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:34.059422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.059857 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.059861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:34.059950 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:34.059978 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:16:34.061781 (+ 1803us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:16:34.063628 (+ 1847us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.127.190:36129: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":1644,"thread_start_us":114,"threads_started":1}
I20260924 07:16:34.064400 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21198
E20260924 07:16:34.072561 21295 webserver.cc:533] Webserver: sslize failed:
E20260924 07:16:34.072646 21295 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T07:16:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T07:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:34Z Disabled control of system clock
I20260924 07:16:34.111433 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:46267
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:41843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:46267
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/testchainca.cert with env {}
W20260924 07:16:34.221176 21299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:34.221449 21299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:34.221524 21299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:34.225504 21299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:34.225555 21299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:34.225570 21299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:34.225636 21299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:34.225672 21299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:16:34.225705 21299 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:16:34.225755 21299 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 07:16:34.231122 21299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:46267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:34.232191 21299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:34.233256 21299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:34.241660 21305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:34.241714 21299 server_base.cc:1034] running on GCE node
W20260924 07:16:34.241660 21307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:34.241668 21304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:34.242487 21299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:34.243578 21299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:34.244773 21299 hybrid_clock.cc:648] HybridClock initialized: now 1790234194244761 us; error 25 us; skew 500 ppm
I20260924 07:16:34.246742 21299 webserver.cc:460] Webserver started at http://127.15.127.190:46789/ using document root <none> and password file <none>
I20260924 07:16:34.247325 21299 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:34.247421 21299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:34.247659 21299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:34.249823 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "1cecd365603b4be093cb486d1335a253"
format_stamp: "Formatted at 2026-09-24 07:16:34 on dist-test-slave-zx46"
server_key: "bbd3c820ce4e3b2a91ca1969ff943ad2"
server_key_iv: "c455b66bda4743a408c38e5483f23406"
server_key_version: "encryptionkey@0"
I20260924 07:16:34.250413 21299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "1cecd365603b4be093cb486d1335a253"
format_stamp: "Formatted at 2026-09-24 07:16:34 on dist-test-slave-zx46"
server_key: "bbd3c820ce4e3b2a91ca1969ff943ad2"
server_key_iv: "c455b66bda4743a408c38e5483f23406"
server_key_version: "encryptionkey@0"
I20260924 07:16:34.254173 21299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:34.256600 21313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:34.257862 21299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:34.258015 21299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "1cecd365603b4be093cb486d1335a253"
format_stamp: "Formatted at 2026-09-24 07:16:34 on dist-test-slave-zx46"
server_key: "bbd3c820ce4e3b2a91ca1969ff943ad2"
server_key_iv: "c455b66bda4743a408c38e5483f23406"
server_key_version: "encryptionkey@0"
I20260924 07:16:34.258145 21299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:34.277244 21299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:34.277978 21299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:34.278189 21299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:34.284987 21299 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:46267
I20260924 07:16:34.284991 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:46267 every 8 connection(s)
I20260924 07:16:34.286000 21299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:34.287119 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21299
I20260924 07:16:34.287258 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:34.287581 15870 external_mini_cluster.cc:1428] Setting key 91f9e20ae4641100bbe03343d5be10f8
I20260924 07:16:34.289299 21366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:34.295013 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253: Bootstrap starting.
I20260924 07:16:34.296639 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:34.288921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43612 (local address 127.15.127.190:46267)
0924 07:16:34.289642 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:34.289658 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:34.289693 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:16:34.290320 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:34.290337 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:34.290730 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:34.291071 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:34.291491 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.292434 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.293379 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.293846 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.294034 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:34.294925 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:34.294929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:34.294942 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:16:34.295378 (+ 436us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:34.295432 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:16:34.295452 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:34.295557 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:34.295695 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 07:16:34.295955 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":130,"threads_started":1}
I20260924 07:16:34.297477 21366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:34.298211 21366 log.cc:824] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:34.299490 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:16:34.300232 21366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253: No bootstrap required, opened a new log
I20260924 07:16:34.303634 21366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cecd365603b4be093cb486d1335a253" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 46267 } }
I20260924 07:16:34.303864 21366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:34.303946 21366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cecd365603b4be093cb486d1335a253, State: Initialized, Role: FOLLOWER
I20260924 07:16:34.304371 21366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cecd365603b4be093cb486d1335a253" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 46267 } }
I20260924 07:16:34.304515 21366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:34.304603 21366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:34.304728 21366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:34.305672 21366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cecd365603b4be093cb486d1335a253" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 46267 } }
I20260924 07:16:34.306018 21366 leader_election.cc:302] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cecd365603b4be093cb486d1335a253; no voters:
I20260924 07:16:34.306324 21366 leader_election.cc:288] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:34.306458 21371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:34.306710 21371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [term 1 LEADER]: Becoming Leader. State: Replica: 1cecd365603b4be093cb486d1335a253, State: Running, Role: LEADER
I20260924 07:16:34.307030 21371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cecd365603b4be093cb486d1335a253" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 46267 } }
I20260924 07:16:34.307626 21366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:34.308509 21372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cecd365603b4be093cb486d1335a253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cecd365603b4be093cb486d1335a253" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 46267 } } }
I20260924 07:16:34.308665 21372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:34.309043 21380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:34.308985 21373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cecd365603b4be093cb486d1335a253. Latest consensus state: current_term: 1 leader_uuid: "1cecd365603b4be093cb486d1335a253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cecd365603b4be093cb486d1335a253" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 46267 } } }
I20260924 07:16:34.309092 21373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:34.311944 21380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:34.317116 21380 catalog_manager.cc:1347] Generated new cluster ID: 2656345cc5d44b3f8e10503423956c3e
I20260924 07:16:34.317191 21380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:34.330163 21380 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:34.331084 21380 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:34.335016 21380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cecd365603b4be093cb486d1335a253: Generated new TSK 0
I20260924 07:16:34.335741 21380 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:16:34.370769 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:34.344357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.15.127.190:46267)
0924 07:16:34.344474 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:34.344478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:34.344646 (+ 168us) server_negotiation.cc:407] Connection header received
0924 07:16:34.344807 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:34.344809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:34.344856 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:34.344949 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:16:34.345252 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.345773 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.346989 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.347201 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.347275 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:34.347604 (+ 329us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:16:34.370367 (+ 22763us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:16:34.370648 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43618: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 07:16:34.370877 21369 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43618: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 07:16:34.371011 21395 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:34.344188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:46267 (local address 127.0.0.1:43618)
0924 07:16:34.344537 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:34.344552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:34.344675 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:34.344972 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:34.344976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:34.344983 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:16:34.345134 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:34.345150 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.345897 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.345901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:34.346872 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:34.346882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:34.347316 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:34.347320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:34.347412 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:34.347440 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:16:34.370612 (+ 23172us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:16:34.370826 (+ 214us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.127.190:46267: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":133,"threads_started":1}
I20260924 07:16:34.371677 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21299
2026-09-24T07:16:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (287 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:34 dist-test-slave-zx46 krb5kdc[21407](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:34 dist-test-slave-zx46 krb5kdc[21407](info): set up 2 sockets
Sep 24 07:16:34 dist-test-slave-zx46 krb5kdc[21407](info): commencing operation
krb5kdc: starting...
W20260924 07:16:35.693521 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:35 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:35.816735 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:42647
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:42647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:35.936897 21420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:35.937263 21420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:35.937342 21420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:35.941588 21420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:35.941656 21420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:35.941675 21420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:35.941728 21420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:35.941761 21420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:35.947430 21420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:42647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:42647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:35.948773 21420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:35.950168 21420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:35.959167 21426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:35.959214 21425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:35.959195 21428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:35.959681 21420 server_base.cc:1034] running on GCE node
I20260924 07:16:35.960228 21420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:35.961424 21420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:35.962662 21420 hybrid_clock.cc:648] HybridClock initialized: now 1790234195962617 us; error 49 us; skew 500 ppm
Sep 24 07:16:35 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234195, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:35.965700 21420 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:35.966679 21420 webserver.cc:460] Webserver started at http://127.15.127.190:46555/ using document root <none> and password file <none>
I20260924 07:16:35.967281 21420 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:35.967381 21420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:35.967622 21420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:35.969907 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "18eca92b8a694caaa06c5f04c869de44"
format_stamp: "Formatted at 2026-09-24 07:16:35 on dist-test-slave-zx46"
server_key: "0e4ee13120578c16a2616e87ddd3c342"
server_key_iv: "202ff15991aef803a8b7858ae89e4f25"
server_key_version: "encryptionkey@0"
I20260924 07:16:35.970508 21420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "18eca92b8a694caaa06c5f04c869de44"
format_stamp: "Formatted at 2026-09-24 07:16:35 on dist-test-slave-zx46"
server_key: "0e4ee13120578c16a2616e87ddd3c342"
server_key_iv: "202ff15991aef803a8b7858ae89e4f25"
server_key_version: "encryptionkey@0"
I20260924 07:16:35.974479 21420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:16:35.977121 21435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:35.978372 21420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:35.978523 21420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "18eca92b8a694caaa06c5f04c869de44"
format_stamp: "Formatted at 2026-09-24 07:16:35 on dist-test-slave-zx46"
server_key: "0e4ee13120578c16a2616e87ddd3c342"
server_key_iv: "202ff15991aef803a8b7858ae89e4f25"
server_key_version: "encryptionkey@0"
I20260924 07:16:35.978658 21420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:35.996384 21420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:36.000109 21420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:36.000332 21420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:36.007818 21420 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:42647
I20260924 07:16:36.007830 21487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:42647 every 8 connection(s)
I20260924 07:16:36.008829 21420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:36.011677 21488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:36.013618 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21420
I20260924 07:16:36.013777 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:36.014055 15870 external_mini_cluster.cc:1428] Setting key 2464cb1b0a7da63c884b44adf7f9e968
I20260924 07:16:36.019080 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44: Bootstrap starting.
I20260924 07:16:36.022047 21488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:36.022950 21488 log.cc:824] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:36.025401 21488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44: No bootstrap required, opened a new log
I20260924 07:16:36.028874 21488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18eca92b8a694caaa06c5f04c869de44" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42647 } }
I20260924 07:16:36.029208 21488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:36.029294 21488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18eca92b8a694caaa06c5f04c869de44, State: Initialized, Role: FOLLOWER
I20260924 07:16:36.029817 21488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18eca92b8a694caaa06c5f04c869de44" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42647 } }
I20260924 07:16:36.030018 21488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:36.030097 21488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:36.030226 21488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:36.030965 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.015608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56700 (local address 127.15.127.190:42647)
0924 07:16:36.016233 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.016253 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.016299 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:16:36.017070 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.017096 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.017538 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.017920 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.019123 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.020623 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.022082 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.022832 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.023435 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.025972 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.025978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.025992 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.026021 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.027814 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.028277 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.028287 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.028292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.028403 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.028705 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.028709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.028711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.029186 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.029423 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.029720 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.030117 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":137,"threads_started":1}
I20260924 07:16:36.031226 21488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18eca92b8a694caaa06c5f04c869de44" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42647 } }
I20260924 07:16:36.031586 21488 leader_election.cc:302] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18eca92b8a694caaa06c5f04c869de44; no voters:
I20260924 07:16:36.031904 21488 leader_election.cc:288] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:36.032034 21493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:36.032305 21493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [term 1 LEADER]: Becoming Leader. State: Replica: 18eca92b8a694caaa06c5f04c869de44, State: Running, Role: LEADER
I20260924 07:16:36.032600 21493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18eca92b8a694caaa06c5f04c869de44" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42647 } }
I20260924 07:16:36.033285 21488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:36.034391 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18eca92b8a694caaa06c5f04c869de44. Latest consensus state: current_term: 1 leader_uuid: "18eca92b8a694caaa06c5f04c869de44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18eca92b8a694caaa06c5f04c869de44" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42647 } } }
I20260924 07:16:36.034533 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:36.034852 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18eca92b8a694caaa06c5f04c869de44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18eca92b8a694caaa06c5f04c869de44" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42647 } } }
I20260924 07:16:36.034946 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:36.034966 21502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:36.037832 21502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:36.043861 21502 catalog_manager.cc:1347] Generated new cluster ID: 72221f31b32c46439f9f13115b2f40e4
I20260924 07:16:36.043939 21502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:36.054968 21502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:36.056221 21502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:36.062464 21502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18eca92b8a694caaa06c5f04c869de44: Generated new TSK 0
I20260924 07:16:36.063206 21502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:36.091379 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42647
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:36.204874 21514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:36.205210 21514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:36.205289 21514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:36.209414 21514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:36.209504 21514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:36.209599 21514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:36.214886 21514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:36.216023 21514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:36.217202 21514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:36.227058 21520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:36.227123 21514 server_base.cc:1034] running on GCE node
W20260924 07:16:36.227058 21519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:36.227058 21522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:36.227808 21514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:36.228502 21514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:36.229688 21514 hybrid_clock.cc:648] HybridClock initialized: now 1790234196229669 us; error 33 us; skew 500 ppm
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.237347 21514 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:36.238317 21514 webserver.cc:460] Webserver started at http://127.15.127.129:35645/ using document root <none> and password file <none>
I20260924 07:16:36.238921 21514 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:36.239015 21514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:36.239260 21514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:36.241493 21514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "79db4188c87e4c58ae0bee712354aaca"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "8ef626e1a5b85526ecbd5b69dce9ac1b"
server_key_iv: "b903717865f68c4f1dfc23755e067cb0"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.242089 21514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "79db4188c87e4c58ae0bee712354aaca"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "8ef626e1a5b85526ecbd5b69dce9ac1b"
server_key_iv: "b903717865f68c4f1dfc23755e067cb0"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.246133 21514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:16:36.248678 21529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.250114 21514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:36.250281 21514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "79db4188c87e4c58ae0bee712354aaca"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "8ef626e1a5b85526ecbd5b69dce9ac1b"
server_key_iv: "b903717865f68c4f1dfc23755e067cb0"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.250411 21514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:36.267652 21514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:36.271054 21514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:36.271275 21514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:36.271888 21514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:36.272748 21514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:36.272830 21514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.272925 21514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:36.272976 21514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.284255 21514 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:46225
I20260924 07:16:36.284276 21642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:46225 every 8 connection(s)
I20260924 07:16:36.285282 21514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:36.287885 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21514
I20260924 07:16:36.288034 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:36.288367 15870 external_mini_cluster.cc:1428] Setting key a4dc0ccb8f927f0cc6977143f6c38631
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:36.301381 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.287212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:55237 (local address 127.15.127.190:42647)
0924 07:16:36.287361 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.287366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.288419 (+ 1053us) server_negotiation.cc:407] Connection header received
0924 07:16:36.289931 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.289933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.289975 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.290042 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.291719 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.292599 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.294401 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.294720 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.294806 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.297765 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.297768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.297771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.297794 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.299299 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.299896 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.299899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.299900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.299984 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.300311 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.300313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.300314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.300490 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.300562 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.301149 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.301256 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:16:36.302484 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.287715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:42647 (local address 127.15.127.129:55237)
0924 07:16:36.288228 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:36.288291 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:36.289700 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:36.290255 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:36.290265 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:36.290808 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:36.291529 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:36.291553 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.292743 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.292745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:36.294272 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:36.294287 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.294889 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.294892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:36.295115 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.295795 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:36.295821 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:36.297582 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:36.299423 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:36.299430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:36.299448 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:36.299781 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:36.300070 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:36.300073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:36.300074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:36.300224 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:36.300572 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:36.300577 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:36.300831 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 07:16:36.301076 (+ 245us) client_negotiation.cc:240] Negotiation successful
0924 07:16:36.301377 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":116,"threads_started":1}
I20260924 07:16:36.303960 21643 heartbeater.cc:347] Connected to a master server at 127.15.127.190:42647
I20260924 07:16:36.304253 21643 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:36.304817 21643 heartbeater.cc:511] Master 127.15.127.190:42647 requested a full tablet report, sending...
I20260924 07:16:36.306550 21452 ts_manager.cc:194] Registered new tserver with Master: 79db4188c87e4c58ae0bee712354aaca (127.15.127.129:46225)
I20260924 07:16:36.308322 21452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:55237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:36.320138 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42647
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:36.447706 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:36.448010 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:36.448065 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:36.452327 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:36.452404 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:36.452464 21648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:36.458360 21648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:36.459537 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:36.460625 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:36.470999 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:36.471020 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:36.470999 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:36.471288 21648 server_base.cc:1034] running on GCE node
I20260924 07:16:36.471798 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:36.472482 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:36.473641 21648 hybrid_clock.cc:648] HybridClock initialized: now 1790234196473624 us; error 32 us; skew 500 ppm
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.476445 21648 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:36.477512 21648 webserver.cc:460] Webserver started at http://127.15.127.130:43847/ using document root <none> and password file <none>
I20260924 07:16:36.478085 21648 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:36.478150 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:36.478323 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:36.480398 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "196b352d48c4447bb1241de9798aab61"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "3ad5ee085e14e6daec27edff7a3446b9"
server_key_iv: "6edff5a0bb3a9b9d6da32176bb4a681a"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.480952 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "196b352d48c4447bb1241de9798aab61"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "3ad5ee085e14e6daec27edff7a3446b9"
server_key_iv: "6edff5a0bb3a9b9d6da32176bb4a681a"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.484724 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:36.487306 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.488545 21648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:36.488667 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "196b352d48c4447bb1241de9798aab61"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "3ad5ee085e14e6daec27edff7a3446b9"
server_key_iv: "6edff5a0bb3a9b9d6da32176bb4a681a"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.488771 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:36.515846 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:36.519055 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:36.519263 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:36.519896 21648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:36.520747 21648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:36.520818 21648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.520892 21648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:36.520936 21648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.530604 21648 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:35681
I20260924 07:16:36.530624 21776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:35681 every 8 connection(s)
I20260924 07:16:36.531551 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:36.537230 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21648
I20260924 07:16:36.537346 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:36.537631 15870 external_mini_cluster.cc:1428] Setting key 10ffc422743eccf0c60dc7d5501e6c93
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:36.548575 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.533551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:45227 (local address 127.15.127.190:42647)
0924 07:16:36.533709 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.533714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.534816 (+ 1102us) server_negotiation.cc:407] Connection header received
0924 07:16:36.536323 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.536327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.536389 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.536473 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.537914 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.538637 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.540583 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.540957 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.541140 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.544254 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.544262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.544266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.544301 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.546172 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.546748 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.546755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.546757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.546868 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.547264 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.547269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.547272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.547536 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.547645 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.548216 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.548359 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:16:36.549479 21779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.534071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:42647 (local address 127.15.127.130:45227)
0924 07:16:36.534658 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:36.534705 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:36.536055 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:36.536656 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:36.536663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:36.537076 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:36.537754 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:36.537771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.538805 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.538808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:36.540427 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:36.540443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.541301 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.541304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:36.541463 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.542138 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:36.542159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:36.544030 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:36.546320 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:36.546326 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:36.546342 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:36.546629 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:36.546972 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:36.546974 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:36.546976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:36.547131 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:36.547647 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:36.547653 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:36.547948 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 07:16:36.548119 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 07:16:36.548436 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":156,"threads_started":1}
I20260924 07:16:36.550904 21777 heartbeater.cc:347] Connected to a master server at 127.15.127.190:42647
I20260924 07:16:36.551142 21777 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:36.551764 21777 heartbeater.cc:511] Master 127.15.127.190:42647 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:36.552906 21452 ts_manager.cc:194] Registered new tserver with Master: 196b352d48c4447bb1241de9798aab61 (127.15.127.130:35681)
I20260924 07:16:36.553566 21452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:45227
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:36.568849 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42647
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:36307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:36.688747 21782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:36.689190 21782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:36.689284 21782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:36.694208 21782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:36.694278 21782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:36.694379 21782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:36.701156 21782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:36307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:42647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:36.702391 21782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:36.703538 21782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:36.714178 21788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:36.714236 21790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:36.714273 21782 server_base.cc:1034] running on GCE node
W20260924 07:16:36.714182 21787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:36.714949 21782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:36.715642 21782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:36.716835 21782 hybrid_clock.cc:648] HybridClock initialized: now 1790234196716816 us; error 35 us; skew 500 ppm
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.719806 21782 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:36.720715 21782 webserver.cc:460] Webserver started at http://127.15.127.131:36179/ using document root <none> and password file <none>
I20260924 07:16:36.721351 21782 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:36.721419 21782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:36.721601 21782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:36.723701 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "5c6a3ca3015543cabe8cc5eaa2c7db56"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "9fccceabef396485fda11b922c04c5c6"
server_key_iv: "36e6d72138d0b928e9fe539f5f3cb312"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.724238 21782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "5c6a3ca3015543cabe8cc5eaa2c7db56"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "9fccceabef396485fda11b922c04c5c6"
server_key_iv: "36e6d72138d0b928e9fe539f5f3cb312"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.728132 21782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:16:36.730747 21797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.732005 21782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:36.732107 21782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "5c6a3ca3015543cabe8cc5eaa2c7db56"
format_stamp: "Formatted at 2026-09-24 07:16:36 on dist-test-slave-zx46"
server_key: "9fccceabef396485fda11b922c04c5c6"
server_key_iv: "36e6d72138d0b928e9fe539f5f3cb312"
server_key_version: "encryptionkey@0"
I20260924 07:16:36.732228 21782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:36.770617 21782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:36.773736 21782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:36.773895 21782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:36.774449 21782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:36.775254 21782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:36.775301 21782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.775341 21782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:36.775357 21782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:36.784510 21782 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:44389
I20260924 07:16:36.784541 21910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:44389 every 8 connection(s)
I20260924 07:16:36.785502 21782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:36.786134 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21782
I20260924 07:16:36.786278 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:36.786604 15870 external_mini_cluster.cc:1428] Setting key b5e6e481c5134eafd78b31b8062eefec
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:36.800045 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.787425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:52309 (local address 127.15.127.190:42647)
0924 07:16:36.787558 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.787564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.788584 (+ 1020us) server_negotiation.cc:407] Connection header received
0924 07:16:36.790126 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.790129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.790182 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.790259 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.791477 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.792136 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.793449 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.793688 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.793799 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.796369 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.796372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.796374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.796400 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.797839 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.798318 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.798320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.798322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.798405 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.798708 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.798710 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.798711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.798887 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.798961 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.799645 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.799816 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:16:36.800976 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.787939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:42647 (local address 127.15.127.131:52309)
0924 07:16:36.788447 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:36.788492 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:36.789914 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:36.790441 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:36.790448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:36.790820 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:36.791332 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:36.791346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.792260 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.792263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:36.793320 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:36.793329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.793841 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.793845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:36.794023 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.794595 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:36.794611 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:36.796190 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:36.797946 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:36.797951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:36.797965 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:36.798229 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:36.798485 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:36.798487 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:36.798489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:36.798622 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:36.798972 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:36.798977 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:36.799229 (+ 252us) client_negotiation.cc:769] Sending connection context
0924 07:16:36.799529 (+ 300us) client_negotiation.cc:240] Negotiation successful
0924 07:16:36.799951 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":156,"threads_started":1}
I20260924 07:16:36.802126 21911 heartbeater.cc:347] Connected to a master server at 127.15.127.190:42647
I20260924 07:16:36.802353 21911 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:36.802814 21911 heartbeater.cc:511] Master 127.15.127.190:42647 requested a full tablet report, sending...
I20260924 07:16:36.803895 21452 ts_manager.cc:194] Registered new tserver with Master: 5c6a3ca3015543cabe8cc5eaa2c7db56 (127.15.127.131:44389)
I20260924 07:16:36.804530 21452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:52309
I20260924 07:16:36.813392 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/test-admin.keytab.
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.830430 15870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 07:16:36.833593 21918 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:36.831885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.15.127.129:46225)
0924 07:16:36.832197 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.832205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.832235 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:16:36.832319 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.832335 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.832557 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.832747 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.833405 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60500: BlockingRecv error: recv got EOF from 127.0.0.1:60500 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":76,"threads_started":1}
W20260924 07:16:36.833829 15870 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.835094 15870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.836138 15870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:36.846500 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.837216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60514 (local address 127.15.127.129:46225)
0924 07:16:36.837410 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.837415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.837496 (+ 81us) server_negotiation.cc:407] Connection header received
0924 07:16:36.837668 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.837672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.837725 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.837815 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.838409 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.839246 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.840085 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.840489 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.840612 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.843012 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.843023 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.843039 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.843066 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.844555 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.845076 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.845086 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.845090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.845192 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.845480 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.845483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.845485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.845763 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.846016 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.846184 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.846344 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260924 07:16:36.850488 21918 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:36.849549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.15.127.129:46225)
0924 07:16:36.849697 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.849700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.849717 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:16:36.849834 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.849837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.849873 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.849937 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.850370 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60520: BlockingRecv error: recv got EOF from 127.0.0.1:60520 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260924 07:16:36.850520 15870 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.851562 15870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.852774 15870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:36.861860 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.853747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60524 (local address 127.15.127.129:46225)
0924 07:16:36.853901 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.853905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.854059 (+ 154us) server_negotiation.cc:407] Connection header received
0924 07:16:36.854106 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.854108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.854147 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.854228 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.854808 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.855395 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.856189 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.856437 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.856527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.858848 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.858852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.858854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.858880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.860238 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.860653 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.860656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.860657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.860738 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.860978 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.860981 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.860982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.861225 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.861350 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.861532 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.861670 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 07:16:36.864730 21918 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:36.863815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.15.127.129:46225)
0924 07:16:36.863956 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.863960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.863974 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:36.864087 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.864090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.864138 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.864215 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.864583 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60536: BlockingRecv error: recv got EOF from 127.0.0.1:60536 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 07:16:36.864809 15870 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790234163642002-15870-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.865907 15870 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:36.866806 15870 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21407](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:16:36.874954 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:36.867627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.15.127.129:46225)
0924 07:16:36.867768 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:36.867771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:36.867834 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:16:36.867922 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:36.867924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:36.867961 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:36.868024 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:36.868469 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.868964 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.869788 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:36.869989 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:36.870064 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:36.872125 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:36.872128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:36.872131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:36.872157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:36.873618 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.873989 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.873991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.873992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.874059 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:36.874277 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:36.874279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:36.874280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:36.874429 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:36.874540 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:36.874691 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 07:16:36.874775 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:36.876030 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21514
I20260924 07:16:36.882210 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21648
I20260924 07:16:36.887647 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21782
I20260924 07:16:36.893177 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21420
2026-09-24T07:16:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2274 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21931](info): set up 2 sockets
Sep 24 07:16:36 dist-test-slave-zx46 krb5kdc[21931](info): commencing operation
krb5kdc: starting...
W20260924 07:16:37.933403 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab.
Entry for principal oryx/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab.
I20260924 07:16:38.060097 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:43345
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab
--principal=oryx/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:43345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:38.172101 21944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:38.172426 21944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:38.172521 21944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:38.176594 21944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:38.176695 21944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:38.176733 21944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:38.176770 21944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:38.176800 21944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:38.182206 21944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:43345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab
--principal=oryx/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:43345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:38.183334 21944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:38.184383 21944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:38.193058 21949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:38.193061 21952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:38.193203 21944 server_base.cc:1034] running on GCE node
W20260924 07:16:38.193058 21950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:38.193965 21944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:38.195098 21944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:38.196265 21944 hybrid_clock.cc:648] HybridClock initialized: now 1790234198196250 us; error 28 us; skew 500 ppm
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:38.202826 21944 init.cc:377] Logged in from keytab as oryx/127.15.127.190@KRBTEST.COM (short username oryx)
I20260924 07:16:38.203867 21944 webserver.cc:460] Webserver started at http://127.15.127.190:34219/ using document root <none> and password file <none>
I20260924 07:16:38.204543 21944 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:38.204613 21944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:38.204861 21944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:38.207072 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "21b39edfe646474880d6855985f97bc5"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "e49752593ce9f8e873e9c7d4c9465397"
server_key_iv: "6aac4b4f7014e5e157a9f79f970701ea"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.207644 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "21b39edfe646474880d6855985f97bc5"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "e49752593ce9f8e873e9c7d4c9465397"
server_key_iv: "6aac4b4f7014e5e157a9f79f970701ea"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.211434 21944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 07:16:38.213886 21959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.215252 21944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:38.215376 21944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "21b39edfe646474880d6855985f97bc5"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "e49752593ce9f8e873e9c7d4c9465397"
server_key_iv: "6aac4b4f7014e5e157a9f79f970701ea"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.215480 21944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:38.228636 21944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:38.231631 21944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:38.231858 21944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:38.238989 21944 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:43345
I20260924 07:16:38.239276 22011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:43345 every 8 connection(s)
I20260924 07:16:38.240031 21944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:38.242794 22012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:38.246506 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 21944
I20260924 07:16:38.246622 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:38.246934 15870 external_mini_cluster.cc:1428] Setting key cebd787316c3d2c259c3edfee36c79bd
I20260924 07:16:38.248963 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Bootstrap starting.
I20260924 07:16:38.251853 22012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:38.252825 22012 log.cc:824] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:38.255265 22012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: No bootstrap required, opened a new log
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:38.259532 22012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } }
I20260924 07:16:38.259795 22012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:38.259859 22012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b39edfe646474880d6855985f97bc5, State: Initialized, Role: FOLLOWER
I20260924 07:16:38.260363 22012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } }
I20260924 07:16:38.260490 22012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:38.260576 22012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:38.260694 22012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:38.261723 22012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } }
I20260924 07:16:38.262109 22012 leader_election.cc:302] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21b39edfe646474880d6855985f97bc5; no voters:
I20260924 07:16:38.262492 22012 leader_election.cc:288] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:38.262568 22017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:38.262769 22017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 LEADER]: Becoming Leader. State: Replica: 21b39edfe646474880d6855985f97bc5, State: Running, Role: LEADER
I20260924 07:16:38.263077 22017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } }
I20260924 07:16:38.263409 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.248499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.15.127.190:43345)
0924 07:16:38.248962 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:38.248984 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:38.249063 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:16:38.249928 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:38.249952 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:38.250432 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:38.250812 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:38.251543 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.253152 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.254230 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.254911 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.255075 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.257920 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:38.257926 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:38.257941 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:38.257974 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:38.259900 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.260405 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.260411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.260417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.260596 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.260944 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.260948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.260950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.261439 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:38.261662 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:38.261972 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 07:16:38.262319 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":110,"threads_started":1}
I20260924 07:16:38.263875 22012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:38.264825 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b39edfe646474880d6855985f97bc5. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } }
I20260924 07:16:38.264959 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:38.265905 22026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:38.265944 22018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } }
I20260924 07:16:38.266021 22018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:38.269065 22026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:38.274458 22026 catalog_manager.cc:1347] Generated new cluster ID: 06e2379f12f84b1dab79dbd49b03d188
I20260924 07:16:38.274544 22026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:38.283043 22026 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:38.284020 22026 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:38.296849 22026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Generated new TSK 0
I20260924 07:16:38.297683 22026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab.
Entry for principal oryx/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab.
I20260924 07:16:38.332175 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43345
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:38.444952 22038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:38.445317 22038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:38.445361 22038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:38.449438 22038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:38.449499 22038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:38.449558 22038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:38.455271 22038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:38.456334 22038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:38.457408 22038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:38.467299 22046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:38.467303 22043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:38.467303 22044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:38.467357 22038 server_base.cc:1034] running on GCE node
I20260924 07:16:38.467974 22038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:38.468672 22038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:38.469837 22038 hybrid_clock.cc:648] HybridClock initialized: now 1790234198469819 us; error 30 us; skew 500 ppm
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:38.472849 22038 init.cc:377] Logged in from keytab as oryx/127.15.127.129@KRBTEST.COM (short username oryx)
I20260924 07:16:38.473804 22038 webserver.cc:460] Webserver started at http://127.15.127.129:45785/ using document root <none> and password file <none>
I20260924 07:16:38.474356 22038 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:38.474417 22038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:38.474589 22038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:38.476675 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "6c773f86301f403f83ff3271ace84f56"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "af94b9bdebda2e9f6a6e08efe2053132"
server_key_iv: "e171d245f011b459a00e65b4f85bcf57"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.477305 22038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "6c773f86301f403f83ff3271ace84f56"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "af94b9bdebda2e9f6a6e08efe2053132"
server_key_iv: "e171d245f011b459a00e65b4f85bcf57"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.481302 22038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260924 07:16:38.483907 22053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.485318 22038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:38.485450 22038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "6c773f86301f403f83ff3271ace84f56"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "af94b9bdebda2e9f6a6e08efe2053132"
server_key_iv: "e171d245f011b459a00e65b4f85bcf57"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.485567 22038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:38.508529 22038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:38.511673 22038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:38.511900 22038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:38.512532 22038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:38.513515 22038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:38.513567 22038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.513635 22038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:38.513666 22038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.523437 22038 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:37253
I20260924 07:16:38.523454 22166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:37253 every 8 connection(s)
I20260924 07:16:38.524515 22038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:38.528862 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22038
I20260924 07:16:38.529036 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:38.529381 15870 external_mini_cluster.cc:1428] Setting key 85be9397c1f004b5404422c5c82f1b18
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:38.540325 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.526450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:40951 (local address 127.15.127.190:43345)
0924 07:16:38.526605 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:38.526610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:38.527641 (+ 1031us) server_negotiation.cc:407] Connection header received
0924 07:16:38.529092 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:38.529096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:38.529160 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:38.529266 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:38.530724 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.531484 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.533049 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.533288 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.533374 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.536240 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:38.536243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:38.536246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:38.536271 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:38.537840 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.538519 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.538522 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.538523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.538623 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.539082 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.539085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.539086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.539287 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:38.539358 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:38.540078 (+ 720us) server_negotiation.cc:299] Negotiation successful
0924 07:16:38.540198 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:16:38.541368 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.526951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:40951)
0924 07:16:38.527469 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:38.527517 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:38.528767 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:38.529471 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:38.529481 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:38.529974 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:38.530560 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:38.530574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.531628 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.531631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:38.532868 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:38.532881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.533464 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.533469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:38.533706 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.534414 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:38.534430 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:38.536018 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:38.537976 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:38.537985 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:38.538005 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:38.538412 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:38.538737 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:38.538741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:38.538742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:38.538971 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:38.539419 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:38.539429 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:38.539824 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 07:16:38.540004 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 07:16:38.540299 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":121,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.130@KRBTEST.COM" created.
I20260924 07:16:38.542845 22167 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43345
I20260924 07:16:38.543143 22167 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:38.543772 22167 heartbeater.cc:511] Master 127.15.127.190:43345 requested a full tablet report, sending...
I20260924 07:16:38.545542 21976 ts_manager.cc:194] Registered new tserver with Master: 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253)
I20260924 07:16:38.546911 21976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:40951
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab.
Entry for principal oryx/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab.
I20260924 07:16:38.561153 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43345
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:38.675381 22172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:38.675724 22172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:38.675815 22172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:38.680045 22172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:38.680161 22172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:38.680259 22172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:38.685828 22172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:38.686985 22172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:38.688043 22172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:38.698189 22180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:38.698268 22178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:38.698239 22177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:38.698371 22172 server_base.cc:1034] running on GCE node
I20260924 07:16:38.698832 22172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:38.699481 22172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:38.700649 22172 hybrid_clock.cc:648] HybridClock initialized: now 1790234198700614 us; error 48 us; skew 500 ppm
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:38.703490 22172 init.cc:377] Logged in from keytab as oryx/127.15.127.130@KRBTEST.COM (short username oryx)
I20260924 07:16:38.704332 22172 webserver.cc:460] Webserver started at http://127.15.127.130:33799/ using document root <none> and password file <none>
I20260924 07:16:38.704907 22172 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:38.704969 22172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:38.705295 22172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:38.707647 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "5c315fa90cb245a9b4cfe62af1e739bb"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "9cc4a68002777e39bf5ac291f44b37fa"
server_key_iv: "77907f4f3a0a5f335df2df0d68a24efc"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.708225 22172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "5c315fa90cb245a9b4cfe62af1e739bb"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "9cc4a68002777e39bf5ac291f44b37fa"
server_key_iv: "77907f4f3a0a5f335df2df0d68a24efc"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.712069 22172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:16:38.714689 22187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.715974 22172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:38.716068 22172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "5c315fa90cb245a9b4cfe62af1e739bb"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "9cc4a68002777e39bf5ac291f44b37fa"
server_key_iv: "77907f4f3a0a5f335df2df0d68a24efc"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.716176 22172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:38.729432 22172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:38.732158 22172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:38.732309 22172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:38.732846 22172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:38.733698 22172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:38.733744 22172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.733783 22172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:38.733805 22172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.742832 22172 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:38543
I20260924 07:16:38.742852 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:38543 every 8 connection(s)
I20260924 07:16:38.743770 22172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:38.748087 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22172
I20260924 07:16:38.748216 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:38.748543 15870 external_mini_cluster.cc:1428] Setting key b6ee8caa285d54139570e8bbde611dd0
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:16:38.759512 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.745729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:54617 (local address 127.15.127.190:43345)
0924 07:16:38.745874 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:38.745878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:38.746753 (+ 875us) server_negotiation.cc:407] Connection header received
0924 07:16:38.748303 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:38.748307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:38.748367 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:38.748428 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:38.749967 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.750720 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.751879 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.752281 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.752401 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.755455 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:38.755459 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:38.755461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:38.755486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:38.756952 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.757733 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.757736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.757737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.757812 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.758272 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.758275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.758276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.758465 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:38.758538 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:38.759281 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 07:16:38.759390 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:38.760591 22303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.746144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:54617)
0924 07:16:38.746596 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:38.746646 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:38.747987 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:38.748847 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:38.748856 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:38.749326 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:38.749817 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:38.749831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.750857 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.750860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:38.751754 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:38.751763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.752435 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.752437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:38.752583 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.753306 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:38.753327 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:38.755249 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:38.757156 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:38.757165 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:38.757184 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:38.757626 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:38.757913 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:38.757917 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:38.757920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:38.758149 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:38.758577 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:38.758593 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:38.758990 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 07:16:38.759255 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 07:16:38.759554 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:38.761909 22301 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43345
I20260924 07:16:38.762199 22301 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:38.762847 22301 heartbeater.cc:511] Master 127.15.127.190:43345 requested a full tablet report, sending...
I20260924 07:16:38.764099 21976 ts_manager.cc:194] Registered new tserver with Master: 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543)
I20260924 07:16:38.764853 21976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:54617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab.
Entry for principal oryx/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab.
I20260924 07:16:38.777872 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43345
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:38.888893 22306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:38.889248 22306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:38.889324 22306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:38.893427 22306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:38.893494 22306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:38.893555 22306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:38.898806 22306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:38.899839 22306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:38.900800 22306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:38.910640 22312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:38.910692 22306 server_base.cc:1034] running on GCE node
W20260924 07:16:38.910651 22314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:38.910634 22311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:38.911274 22306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:38.911864 22306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:38.913075 22306 hybrid_clock.cc:648] HybridClock initialized: now 1790234198913051 us; error 31 us; skew 500 ppm
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:38.915877 22306 init.cc:377] Logged in from keytab as oryx/127.15.127.131@KRBTEST.COM (short username oryx)
I20260924 07:16:38.916805 22306 webserver.cc:460] Webserver started at http://127.15.127.131:43047/ using document root <none> and password file <none>
I20260924 07:16:38.917418 22306 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:38.917480 22306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:38.917702 22306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:38.919942 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "930a85194c5f41dbb7fdad053539fbce"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "36ea3fd5ede43e5d073a219e0f9c2a19"
server_key_iv: "2c885c8f1b74b4a31ab3bc3ad81ed2f6"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.920552 22306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "930a85194c5f41dbb7fdad053539fbce"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "36ea3fd5ede43e5d073a219e0f9c2a19"
server_key_iv: "2c885c8f1b74b4a31ab3bc3ad81ed2f6"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.924348 22306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:16:38.926774 22321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.928011 22306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:38.928174 22306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "930a85194c5f41dbb7fdad053539fbce"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "36ea3fd5ede43e5d073a219e0f9c2a19"
server_key_iv: "2c885c8f1b74b4a31ab3bc3ad81ed2f6"
server_key_version: "encryptionkey@0"
I20260924 07:16:38.928294 22306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:38.950044 22306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:38.953509 22306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:38.953717 22306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:38.954342 22306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:38.955119 22306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:38.955164 22306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.955204 22306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:38.955219 22306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:38.964481 22306 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:34001
I20260924 07:16:38.964527 22434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:34001 every 8 connection(s)
I20260924 07:16:38.965478 22306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:38.974385 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22306
I20260924 07:16:38.974516 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:38.974835 15870 external_mini_cluster.cc:1428] Setting key 1cc015ffc7ce14772d100bb425b60033
Sep 24 07:16:38 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:38.979983 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.967577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:41961 (local address 127.15.127.190:43345)
0924 07:16:38.967761 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:38.967766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:38.968700 (+ 934us) server_negotiation.cc:407] Connection header received
0924 07:16:38.970056 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:38.970060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:38.970109 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:38.970189 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:38.971446 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.971994 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.973163 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.973378 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.973462 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.976261 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:38.976264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:38.976267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:38.976302 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:38.977885 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.978419 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.978422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.978424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.978512 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:38.978871 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:38.978876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:38.978878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:38.979078 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:38.979195 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:38.979696 (+ 501us) server_negotiation.cc:299] Negotiation successful
0924 07:16:38.979823 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 07:16:38.980839 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:38.968063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:41961)
0924 07:16:38.968535 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:38.968581 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:38.969841 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:38.970382 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:38.970390 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:38.970766 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:38.971282 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:38.971302 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.972112 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.972115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:38.973060 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:38.973069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:38.973509 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:38.973512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:38.973651 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:38.974284 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:38.974304 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:38.976091 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:38.978017 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:38.978024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:38.978041 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:38.978313 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:38.978610 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:38.978613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:38.978615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:38.978774 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:38.979192 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:38.979200 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:38.979457 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 07:16:38.979663 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 07:16:38.979998 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":117,"threads_started":1}
I20260924 07:16:38.981998 22435 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43345
I20260924 07:16:38.982262 22435 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:38.982767 22435 heartbeater.cc:511] Master 127.15.127.190:43345 requested a full tablet report, sending...
I20260924 07:16:38.983835 21976 ts_manager.cc:194] Registered new tserver with Master: 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001)
I20260924 07:16:38.984409 21976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:41961
I20260924 07:16:38.988956 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.189.keytab.
Entry for principal oryx/127.15.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.189.keytab.
I20260924 07:16:39.015650 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:44671
--webserver_interface=127.15.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.189.keytab
--principal=oryx/127.15.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:43345,127.15.127.189:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:39.127327 22440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:39.127609 22440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:39.127655 22440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:39.131790 22440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:39.131858 22440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:39.131877 22440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:39.131920 22440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:39.131951 22440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:39.137332 22440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:43345,127.15.127.189:44671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.189.keytab
--principal=oryx/127.15.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
W20260924 07:16:39.137576 22440 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.127.190:43345,127.15.127.189:44671'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 07:16:39.138525 22440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:39.139639 22440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:39.148169 22445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.148276 22446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.148283 22440 server_base.cc:1034] running on GCE node
W20260924 07:16:39.148190 22448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.149129 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:39.150249 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:39.151422 22440 hybrid_clock.cc:648] HybridClock initialized: now 1790234199151396 us; error 43 us; skew 500 ppm
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:39.154498 22440 init.cc:377] Logged in from keytab as oryx/127.15.127.189@KRBTEST.COM (short username oryx)
I20260924 07:16:39.155431 22440 webserver.cc:460] Webserver started at http://127.15.127.189:35937/ using document root <none> and password file <none>
I20260924 07:16:39.155985 22440 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.156046 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.156234 22440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:39.158428 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/instance:
uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "fc7788a61b59a5d8d3d2433c7b114533"
server_key_iv: "3b179c63c1331cd8d3ac742201101a80"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.158957 22440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal/instance:
uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "fc7788a61b59a5d8d3d2433c7b114533"
server_key_iv: "3b179c63c1331cd8d3ac742201101a80"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.162730 22440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:39.165390 22455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.166697 22440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:16:39.166824 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "fc7788a61b59a5d8d3d2433c7b114533"
server_key_iv: "3b179c63c1331cd8d3ac742201101a80"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.166930 22440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.202037 22440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:39.205506 22440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:39.205736 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:39.212590 22440 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.189:44671
I20260924 07:16:39.212894 22507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.189:44671 every 8 connection(s)
I20260924 07:16:39.213667 22440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
I20260924 07:16:39.216566 22508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:39.220641 22508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:16:39.222450 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22440
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.189@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
I20260924 07:16:39.235028 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.222726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.15.127.190:43345)
0924 07:16:39.222864 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.222868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.223971 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 07:16:39.224925 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.224927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.224977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.225120 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.226051 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.226572 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.227718 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.227930 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.228010 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.230877 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.230883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.230887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.230921 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.232663 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.233263 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.233265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.233266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.233338 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.233717 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.233721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.233723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.234003 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.234118 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.234709 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.234841 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:16:39.236058 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.223313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.15.127.189:44671)
0924 07:16:39.223899 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.223912 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.223952 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:16:39.224702 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.224727 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.224970 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.225192 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.225886 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.226868 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.227655 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.228085 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.228344 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.231377 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.231387 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.231404 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.231430 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.233456 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.233914 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.233918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.233919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.233983 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.234273 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.234276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.234278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.234522 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.234970 (+ 448us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.235083 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.235202 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":501,"thread_start_us":142,"threads_started":1}
I20260924 07:16:39.236058 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.223183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:57514)
0924 07:16:39.223793 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.223854 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.224702 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.225160 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.225165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.225616 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.225899 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.225914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.226702 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.226706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.227556 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.227564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.228075 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.228079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.228340 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.229139 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.229157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.230677 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.232771 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.232780 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.232796 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.233165 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.233428 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.233431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.233433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.233626 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.234126 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.234131 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.234436 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.234644 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.234972 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":95,"spinlock_wait_cycles":56704,"thread_start_us":155,"threads_started":1}
I20260924 07:16:39.237465 22508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.189@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
I20260924 07:16:39.245481 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.238031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.15.127.189:44671)
0924 07:16:39.238139 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.238143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.238157 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:39.238254 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.238260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.238305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.238396 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.238904 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.239384 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.240108 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.240328 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.240435 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.242518 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.242522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.242525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.242550 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.243895 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.244315 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.244321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.244323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.244392 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.244650 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.244653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.244655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.244811 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.244913 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.245204 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.245342 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":49}
I20260924 07:16:39.245481 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.237974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57674)
0924 07:16:39.238087 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.238103 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.238177 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.238402 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.238405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.238670 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.238752 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.238757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.239481 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.239487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.239992 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.239998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.240431 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.240434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.240587 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.241069 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.241088 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.242390 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.244042 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.244045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.244047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.244224 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.244484 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.244487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.244489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.244563 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.244902 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.244905 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.244984 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.245161 (+ 177us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.245296 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":40}
I20260924 07:16:39.249389 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Bootstrap starting.
I20260924 07:16:39.252132 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:39.252909 22508 log.cc:824] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:39.254983 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: No bootstrap required, opened a new log
I20260924 07:16:39.258121 22508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } }
I20260924 07:16:39.258335 22508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:39.258420 22508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5f661bd8dc40e0972223ed165e9d4e, State: Initialized, Role: FOLLOWER
I20260924 07:16:39.258852 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } }
I20260924 07:16:39.259354 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } } }
I20260924 07:16:39.259490 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:39.260102 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:16:39.265307 22525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:16:39.265391 22525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:16:39.267220 22440 master_runner.cc:428] Detected that this master 2f5f661bd8dc40e0972223ed165e9d4e is joining an existing cluster
I20260924 07:16:39.267297 22440 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.127.189:44671
I20260924 07:16:39.278728 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.270104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57684 (local address 127.15.127.189:44671)
0924 07:16:39.270232 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.270238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.270515 (+ 277us) server_negotiation.cc:407] Connection header received
0924 07:16:39.270677 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.270680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.270726 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.270811 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.271567 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.272116 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.273323 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.273583 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.273634 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.275215 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.275219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.275221 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.275271 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.277037 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277492 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.277495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.277497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.277575 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277872 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.277875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.277877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.278070 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.278138 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.278430 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.278544 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:16:39.279125 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.269709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57520 (local address 127.15.127.190:43345)
0924 07:16:39.269857 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.269863 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.270462 (+ 599us) server_negotiation.cc:407] Connection header received
0924 07:16:39.270625 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.270631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.270704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.270747 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.271468 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.272338 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.273438 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.273799 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.273909 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.275324 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.275327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.275330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.275355 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.277275 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277743 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.277747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.277749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.277833 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.278110 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.278115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.278116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.278303 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.278410 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.278664 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.278773 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:16:39.279143 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.269563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:57520)
0924 07:16:39.270364 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.270380 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.270492 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.270909 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.270914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.271197 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.271334 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.271342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.272452 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.272455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.273328 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.273335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.273923 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.273926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.274050 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.274485 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.274503 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.275183 (+ 680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.277375 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.277382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.277618 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.277931 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277934 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.277936 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.278019 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.278399 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.278402 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.278497 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.278602 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.278759 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":128,"threads_started":1}
I20260924 07:16:39.279148 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.270001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57684)
0924 07:16:39.270448 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.270461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.270552 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.270845 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.270851 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.271197 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.271458 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.271464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.272229 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.272235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.273204 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.273214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.273757 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.273760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.273814 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.274607 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.274625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.275079 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.277135 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277139 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.277141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.277392 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.277676 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.277679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.277684 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.277775 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.278195 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.278197 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.278282 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.278867 (+ 585us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.278989 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":90,"threads_started":1}
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.190@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
I20260924 07:16:39.295912 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.284020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57698)
0924 07:16:39.284399 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.284449 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.284580 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.284886 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.284890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.285332 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.285448 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.285454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.286433 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.286435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.287058 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.287066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.287602 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.287604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.287692 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.288284 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.288297 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.291765 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.293893 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.293908 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.293919 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.294265 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.294573 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.294577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.294582 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.294705 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.295183 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.295191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.295325 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.295486 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.295720 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":129,"threads_started":1}
I20260924 07:16:39.295931 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.284060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.15.127.189:44671)
0924 07:16:39.284181 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.284185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.284528 (+ 343us) server_negotiation.cc:407] Connection header received
0924 07:16:39.284710 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.284713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.284765 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.284846 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.285557 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.286327 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.287164 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.287497 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.287605 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.291918 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.291922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.291926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.291950 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.293759 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.294374 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.294378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.294380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.294465 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.294809 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.294812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.294814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.295033 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.295123 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.295581 (+ 458us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.295746 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:39.297676 21976 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.127.189:44671: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
member_type: NON_VOTER
last_known_addr {
host: "127.15.127.189"
port: 44671
}
attrs {
promote: true
}
}
dest_uuid: "21b39edfe646474880d6855985f97bc5"
cas_config_opid_index: -1
I20260924 07:16:39.298403 21976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.15.127.188@KRBTEST.COM; defaulting to no policy
I20260924 07:16:39.301185 22482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.188@KRBTEST.COM" created.
I20260924 07:16:39.301331 22536 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2f5f661bd8dc40e0972223ed165e9d4e (127.15.127.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } }
I20260924 07:16:39.302657 22482 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Refusing update from remote peer 21b39edfe646474880d6855985f97bc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 07:16:39.303449 22536 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.127.189:44671
I20260924 07:16:39.303696 22536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:39.303799 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } }
I20260924 07:16:39.303910 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:39.303923 22440 master.cc:537] Master@127.15.127.189:44671 shutting down...
W20260924 07:16:39.305876 21963 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 07:16:39.306869 21963 consensus_peers.cc:487] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 -> Peer 2f5f661bd8dc40e0972223ed165e9d4e (127.15.127.189:44671): Couldn't send request to peer 2f5f661bd8dc40e0972223ed165e9d4e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:16:39.309451 22440 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:39.309808 22440 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:39.309948 22440 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: stopping tablet replica
I20260924 07:16:39.315100 22440 master.cc:552] Master@127.15.127.189:44671 shutdown complete.
I20260924 07:16:39.315191 22440 master_runner.cc:305] Clearing existing system tablet
I20260924 07:16:39.315421 22440 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.315492 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.316711 22440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:16:39.317960 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.318225 22440 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.318413 22440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:39.318492 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "fc7788a61b59a5d8d3d2433c7b114533"
server_key_iv: "3b179c63c1331cd8d3ac742201101a80"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.320924 22440 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.188.keytab.
Entry for principal oryx/127.15.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.188.keytab.
I20260924 07:16:39.322407 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:45171
--webserver_interface=127.15.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.188.keytab
--principal=oryx/127.15.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:16:39.323412 22440 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:16:39.323531 22440 log.cc:1196] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 07:16:39.323832 22440 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Deleting consensus metadata
I20260924 07:16:39.324028 22440 master_runner.cc:315] Copying system tablet from 127.15.127.190:43345
I20260924 07:16:39.325474 22440 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.127.190:43345
I20260924 07:16:39.340840 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.326120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:57524)
0924 07:16:39.331850 (+ 5730us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.331871 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.332045 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.332407 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.332410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.332658 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.332835 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.332842 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.334094 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.334098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.335024 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.335035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.335649 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.335653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.335757 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.336387 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.336402 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.336861 (+ 459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.339058 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.339062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.339064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.339341 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.339657 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.339660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.339662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.339764 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.340215 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.340218 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.340331 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.340442 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.340615 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5619,"thread_start_us":88,"threads_started":1}
I20260924 07:16:39.340858 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.326243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.15.127.190:43345)
0924 07:16:39.326396 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.326403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.332013 (+ 5610us) server_negotiation.cc:407] Connection header received
0924 07:16:39.332164 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.332170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.332250 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.332376 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.332978 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.333962 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.335153 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.335513 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.335618 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.337106 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.337111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.337114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.337144 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.338935 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.339458 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.339462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.339464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.339556 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.339859 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.339862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.339864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.340104 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.340218 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.340538 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.340689 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:16:39.341522 21996 tablet_copy_service.cc:140] P 21b39edfe646474880d6855985f97bc5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2f5f661bd8dc40e0972223ed165e9d4e ({username='oryx', principal='oryx/127.15.127.189@KRBTEST.COM'} at 127.0.0.1:57524)
I20260924 07:16:39.341737 21996 tablet_copy_service.cc:161] P 21b39edfe646474880d6855985f97bc5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2f5f661bd8dc40e0972223ed165e9d4e at {username='oryx', principal='oryx/127.15.127.189@KRBTEST.COM'} at 127.0.0.1:57524: session id = 2f5f661bd8dc40e0972223ed165e9d4e-00000000000000000000000000000000
I20260924 07:16:39.343591 21996 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:16:39.345477 22440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:16:39.348037 22440 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: tablet copy: Starting download of 0 data blocks...
I20260924 07:16:39.348244 22440 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: tablet copy: Starting download of 1 WAL segments...
I20260924 07:16:39.350319 22440 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:16:39.352330 21996 tablet_copy_service.cc:342] P 21b39edfe646474880d6855985f97bc5: Request end of tablet copy session 2f5f661bd8dc40e0972223ed165e9d4e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.127.189@KRBTEST.COM'} at 127.0.0.1:57524
I20260924 07:16:39.352479 21996 tablet_copy_service.cc:433] P 21b39edfe646474880d6855985f97bc5: ending tablet copy session 2f5f661bd8dc40e0972223ed165e9d4e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2f5f661bd8dc40e0972223ed165e9d4e
I20260924 07:16:39.359977 22440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:39.361899 22551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.361915 22552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.362152 22554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.365191 22440 server_base.cc:1034] running on GCE node
I20260924 07:16:39.365361 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:39.365756 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:39.366912 22440 hybrid_clock.cc:648] HybridClock initialized: now 1790234199366929 us; error 59 us; skew 500 ppm
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:39.368670 22440 init.cc:377] Logged in from keytab as oryx/127.15.127.189@KRBTEST.COM (short username oryx)
I20260924 07:16:39.369226 22440 webserver.cc:460] Webserver started at http://127.15.127.189:36353/ using document root <none> and password file <none>
I20260924 07:16:39.369427 22440 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.369469 22440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.370817 22440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:39.371893 22561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.372191 22440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:39.372274 22440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "fc7788a61b59a5d8d3d2433c7b114533"
server_key_iv: "3b179c63c1331cd8d3ac742201101a80"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.372367 22440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.400192 22440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:39.402530 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:39.407356 22440 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.189:44671
I20260924 07:16:39.407397 22613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.189:44671 every 8 connection(s)
I20260924 07:16:39.407934 22440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
I20260924 07:16:39.409837 22614 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:16:39.410908 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Bootstrap starting.
I20260924 07:16:39.421489 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:16:39.421850 22614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Bootstrap complete.
I20260924 07:16:39.422494 22614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } }
I20260924 07:16:39.422663 22614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f5f661bd8dc40e0972223ed165e9d4e, State: Initialized, Role: LEARNER
I20260924 07:16:39.422794 22614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } }
I20260924 07:16:39.423183 22617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } }
I20260924 07:16:39.423321 22614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:39.423362 22617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: LEARNER
I20260924 07:16:39.426367 22628 catalog_manager.cc:1259] Loaded cluster ID: 06e2379f12f84b1dab79dbd49b03d188
I20260924 07:16:39.426503 22628 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: loading cluster ID for follower catalog manager: success
I20260924 07:16:39.428893 22628 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: acquiring CA information for follower catalog manager: success
I20260924 07:16:39.430318 22628 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:16:39.462308 22544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:39.462647 22544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:39.462728 22544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:39.467170 22544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:39.467252 22544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:39.467290 22544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:39.467346 22544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:39.467383 22544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:39.473110 22544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.188.keytab
--principal=oryx/127.15.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:45171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:39.474355 22544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:39.475759 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:39.484666 22634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.484719 22633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.484705 22636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.484760 22544 server_base.cc:1034] running on GCE node
I20260924 07:16:39.485718 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:39.486799 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:39.487959 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790234199487945 us; error 30 us; skew 500 ppm
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:39.490938 22544 init.cc:377] Logged in from keytab as oryx/127.15.127.188@KRBTEST.COM (short username oryx)
I20260924 07:16:39.491840 22544 webserver.cc:460] Webserver started at http://127.15.127.188:35915/ using document root <none> and password file <none>
I20260924 07:16:39.492403 22544 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.492492 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.492740 22544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:39.494863 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/instance:
uuid: "87db3c4cfe024dadb0995f9bf4f44719"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "790abb9b849869060d79e10ae2ff01fd"
server_key_iv: "3b2ab079de2492b9a5c8481b0e31de08"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.495510 22544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal/instance:
uuid: "87db3c4cfe024dadb0995f9bf4f44719"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "790abb9b849869060d79e10ae2ff01fd"
server_key_iv: "3b2ab079de2492b9a5c8481b0e31de08"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.499287 22544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:39.501698 22643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.503057 22544 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:39.503214 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
uuid: "87db3c4cfe024dadb0995f9bf4f44719"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "790abb9b849869060d79e10ae2ff01fd"
server_key_iv: "3b2ab079de2492b9a5c8481b0e31de08"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.503333 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.526329 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:39.529790 22544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:39.529951 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:39.537644 22544 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.188:45171
I20260924 07:16:39.537930 22695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.188:45171 every 8 connection(s)
I20260924 07:16:39.538904 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
I20260924 07:16:39.540720 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22544
I20260924 07:16:39.542559 22696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:39.546782 22696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:16:39.549433 22167 heartbeater.cc:503] Master 127.15.127.190:43345 was elected leader, sending a full tablet report...
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.127.188@KRBTEST.COM
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.188@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:39.556527 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.542339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.15.127.188:45171)
0924 07:16:39.542881 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.542898 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.542937 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:16:39.543634 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.543654 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.544086 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.544541 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.545389 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.546459 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.547394 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.547895 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.548075 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.551344 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.551353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.551373 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.551401 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.553420 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.553913 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.553918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.553923 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.554063 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.554375 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.554380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.554386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.554871 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.555112 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.555360 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.555659 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":173,"threads_started":1}
I20260924 07:16:39.558837 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.548868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57534 (local address 127.15.127.190:43345)
0924 07:16:39.549045 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.549050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.549342 (+ 292us) server_negotiation.cc:407] Connection header received
0924 07:16:39.549592 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.549596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.549646 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.549734 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.550549 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.551243 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.552123 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.552484 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.552613 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.555063 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.555066 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.555071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.555101 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.556603 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.557491 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.557493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.557495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.557585 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.557900 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.557904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.557906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.558101 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.558212 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.558548 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.558665 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:16:39.558853 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.548629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:57534)
0924 07:16:39.549191 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.549253 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.549456 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.549883 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.549890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.550290 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.550418 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.550424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.551353 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.551356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.552001 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.552009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.552602 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.552605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.552697 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.553343 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.553355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.554893 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.557081 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.557087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.557092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.557362 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.557694 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.557697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.557699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.557792 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.558208 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.558215 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.558324 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.558469 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.558676 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":188,"threads_started":1}
I20260924 07:16:39.559971 22696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.188@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
I20260924 07:16:39.568327 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.560478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57708)
0924 07:16:39.560593 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.560609 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.560697 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.561129 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.561131 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.561273 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.561368 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.561372 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.562240 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.562242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.562777 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.562784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.563252 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.563254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.563332 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.563857 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.563869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.565261 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.566977 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.566980 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.566982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.567177 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.567414 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.567416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.567418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.567491 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.567885 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.567888 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.567977 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.568072 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.568197 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 07:16:39.568356 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.560598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.15.127.189:44671)
0924 07:16:39.560865 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.560868 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.560880 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:39.560918 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.560920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.560960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.561110 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.561464 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.562147 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.562885 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.563154 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.563238 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.565398 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.565401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.565403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.565426 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.566874 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.567262 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.567265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.567266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.567338 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.567565 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.567567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.567568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.567764 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.567842 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.568123 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.568229 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260924 07:16:39.569293 22696 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.188@KRBTEST.COM for oryx/127.15.127.188@KRBTEST.COM
I20260924 07:16:39.577204 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.569842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43440)
0924 07:16:39.569990 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.570006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.570115 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.570366 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.570368 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.570541 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.570606 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.570611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.571344 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.571350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.571863 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.571871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.572309 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.572312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.572461 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.573062 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.573073 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.574333 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.575845 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.575848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.575849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.576037 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.576312 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.576314 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.576316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.576390 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.576719 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.576721 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.576800 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.576879 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.576986 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":54}
I20260924 07:16:39.577700 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.569881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43440 (local address 127.15.127.188:45171)
0924 07:16:39.569991 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.569997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.570073 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:16:39.570240 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.570243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.570276 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.570358 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.570732 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.571242 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.571973 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.572205 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.572311 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.574470 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.574473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.574476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.574496 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.575732 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.576122 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.576125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.576126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.576221 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.576466 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.576468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.576469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.576639 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.576714 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.577497 (+ 783us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.577593 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":42}
I20260924 07:16:39.580598 22696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Bootstrap starting.
I20260924 07:16:39.583009 22696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:39.583777 22696 log.cc:824] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:39.586133 22696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: No bootstrap required, opened a new log
I20260924 07:16:39.590514 22696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } }
I20260924 07:16:39.590775 22696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:39.590862 22696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87db3c4cfe024dadb0995f9bf4f44719, State: Initialized, Role: FOLLOWER
I20260924 07:16:39.591375 22696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } }
I20260924 07:16:39.591871 22702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } } }
I20260924 07:16:39.591988 22702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:39.592651 22696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:16:39.597380 22713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:16:39.597481 22713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:16:39.601150 22544 master_runner.cc:428] Detected that this master 87db3c4cfe024dadb0995f9bf4f44719 is joining an existing cluster
I20260924 07:16:39.601217 22544 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.127.188:45171
I20260924 07:16:39.610101 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22038
I20260924 07:16:39.614812 22719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.603741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:57544)
0924 07:16:39.604052 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.604067 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.604175 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.604788 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.604793 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.605117 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.605286 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.605292 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.606367 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.606370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.608156 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.608166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.609235 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.609241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.609339 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.609829 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.609843 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.610919 (+ 1076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.612976 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.612979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.612981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.613264 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.613590 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.613594 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.613597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.613694 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.614239 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.614242 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.614346 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.614442 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.614578 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":8,"spinlock_wait_cycles":8704,"thread_start_us":112,"threads_started":1}
I20260924 07:16:39.615283 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.604285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.15.127.189:44671)
0924 07:16:39.604414 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.604420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.604517 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:16:39.604694 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.604699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.604752 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.604803 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.605537 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.606490 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.607884 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.608247 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608324 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.611780 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.611784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.611789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.611824 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.613504 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.613949 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.613952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.613953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.614027 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.614353 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.614356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.614357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.614566 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.614640 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.614995 (+ 355us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.615118 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:39.616251 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.603896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.15.127.190:43345)
0924 07:16:39.604031 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.604037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.604449 (+ 412us) server_negotiation.cc:407] Connection header received
0924 07:16:39.604502 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.604506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.604559 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.604657 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.605411 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.606254 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608286 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.608623 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608726 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.611051 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.611055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.611057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.611084 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.612845 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.613385 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.613392 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.613395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.613483 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.613804 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.613809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.613812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.614100 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.614200 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.615299 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.616110 (+ 811us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:39.615844 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.604199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57710)
0924 07:16:39.604440 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.604452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.604576 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.604943 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.604946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.605272 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.605403 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.605411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.606751 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.606754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.607753 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.607763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.608542 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.608675 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.609264 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.609280 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.611627 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.613607 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.613609 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.613611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.613845 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.614139 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.614145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.614150 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.614253 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.614708 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.614711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.614801 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.615497 (+ 696us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.615647 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260924 07:16:39.616993 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:37253
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:43345
--webserver_port=45785
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234198, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.190@KRBTEST.COM for oryx/127.15.127.188@KRBTEST.COM
I20260924 07:16:39.638109 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.620764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43470)
0924 07:16:39.620938 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.620954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.621057 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.621552 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.621555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.621751 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.621834 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.621839 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.622588 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.622593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.623143 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.623151 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.623694 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.623698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.623791 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.624279 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.624293 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.625822 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.636343 (+ 10521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.636350 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.636353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.636671 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.637065 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.637069 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.637070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.637168 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.637557 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.637562 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.637680 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.637764 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.637939 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 07:16:39.638109 22698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.604524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.15.127.188:45171)
0924 07:16:39.604634 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.604638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.605099 (+ 461us) server_negotiation.cc:407] Connection header received
0924 07:16:39.605254 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.605257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.605292 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.605337 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.605833 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.606508 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608370 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.608714 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608771 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.634456 (+ 25685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.634464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.634473 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.634505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.636385 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.636798 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.636801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.636803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.636884 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.637206 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.637212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.637216 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.637445 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.637530 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.637835 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.637939 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":11,"server-negotiator.queue_time_us":37}
I20260924 07:16:39.638779 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.620824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.15.127.188:45171)
0924 07:16:39.621273 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.621282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.621296 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:39.621366 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.621369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.621440 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.621535 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.621953 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.622482 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.623256 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.623573 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.623694 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.625982 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.625986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.625989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.626020 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.636175 (+ 10155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.636798 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.636801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.636803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.636922 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.637280 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.637283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.637285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.637445 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.638416 (+ 971us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.638495 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.638602 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":38272,"thread_start_us":114,"threads_started":1}
I20260924 07:16:39.639245 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.604708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43456)
0924 07:16:39.605025 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.605039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.605164 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.605454 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.605457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.605630 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.605728 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.605733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.606668 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.606672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.608276 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.608283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.608923 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.608927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.609041 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.633636 (+ 24595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.633658 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.634240 (+ 582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.636487 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.636489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.636491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.636702 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.636989 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.636994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.637022 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.637099 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.637600 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.637602 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.637679 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.638727 (+ 1048us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.639078 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":58,"spinlock_wait_cycles":1408,"thread_start_us":77,"threads_started":1}
I20260924 07:16:39.639791 21976 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.127.188:45171: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719"
member_type: NON_VOTER
last_known_addr {
host: "127.15.127.188"
port: 45171
}
attrs {
promote: true
}
}
dest_uuid: "21b39edfe646474880d6855985f97bc5"
cas_config_opid_index: 5
I20260924 07:16:39.640409 21976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } }
I20260924 07:16:39.641724 22538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.642982 22538 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.127.188:45171
I20260924 07:16:39.643169 22538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.643270 22538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:39.643796 22544 master.cc:537] Master@127.15.127.188:45171 shutting down...
W20260924 07:16:39.645332 21963 consensus_peers.cc:487] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 -> Peer 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188:45171): Couldn't send request to peer 87db3c4cfe024dadb0995f9bf4f44719. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:16:39.648449 22544 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:39.648749 22544 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:39.648845 22544 tablet_replica.cc:330] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: stopping tablet replica
I20260924 07:16:39.649972 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.641806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57726)
0924 07:16:39.645498 (+ 3692us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.645514 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.645602 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.646079 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.646082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.646101 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:39.646211 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.646217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.647054 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.647056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.648298 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.648310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.649546 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.649549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.649653 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.649698 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.649775 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.649850 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3608}
I20260924 07:16:39.650396 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.643525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.15.127.189:44671)
0924 07:16:39.643739 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.643745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.645708 (+ 1963us) server_negotiation.cc:407] Connection header received
0924 07:16:39.645759 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.645761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.645849 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.645937 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:39.646331 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.646895 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.648445 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.649093 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.649221 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.649369 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.650209 (+ 840us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.650288 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260924 07:16:39.651304 22588 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 LEARNER]: Refusing update from remote peer 21b39edfe646474880d6855985f97bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:16:39.651774 22536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 07:16:39.654392 22588 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.656900 22617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b39edfe646474880d6855985f97bc5. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.657064 22617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: LEARNER
I20260924 07:16:39.657209 22544 master.cc:552] Master@127.15.127.188:45171 shutdown complete.
I20260924 07:16:39.657179 22617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: NON_VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: true } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.657246 22544 master_runner.cc:305] Clearing existing system tablet
I20260924 07:16:39.657264 22617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: LEARNER
I20260924 07:16:39.657480 22544 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.657518 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.658547 22544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:39.659376 22536 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: attempting to promote NON_VOTER 2f5f661bd8dc40e0972223ed165e9d4e to VOTER
I20260924 07:16:39.659406 22727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.659565 22544 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.659664 22544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:39.659698 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
uuid: "87db3c4cfe024dadb0995f9bf4f44719"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "790abb9b849869060d79e10ae2ff01fd"
server_key_iv: "3b2ab079de2492b9a5c8481b0e31de08"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.659875 22536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } }
I20260924 07:16:39.661516 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.661118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43480)
0924 07:16:39.661356 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.661432 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 07:16:39.661762 22544 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:39.661944 22587 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 LEARNER]: Refusing update from remote peer 21b39edfe646474880d6855985f97bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 07:16:39.662572 22538 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 07:16:39.663719 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.663360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43484)
0924 07:16:39.663585 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.663648 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:45171: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260924 07:16:39.664376 22544 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:16:39.664453 22544 log.cc:1196] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 07:16:39.664700 22544 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Deleting consensus metadata
W20260924 07:16:39.664706 21963 consensus_peers.cc:487] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 -> Peer 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188:45171): Couldn't send request to peer 87db3c4cfe024dadb0995f9bf4f44719. Status: Network error: Client connection negotiation failed: client connection to 127.15.127.188:45171: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:16:39.664943 22544 master_runner.cc:315] Copying system tablet from 127.15.127.190:43345
I20260924 07:16:39.665807 22538 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2f5f661bd8dc40e0972223ed165e9d4e (127.15.127.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.666179 22544 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: tablet copy: Beginning tablet copy session from remote peer at address 127.15.127.190:43345
I20260924 07:16:39.667670 22538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.667783 22538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:39.667951 22538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.668023 22538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:39.668839 22587 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2f5f661bd8dc40e0972223ed165e9d4e (127.15.127.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.670712 22617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.670831 22617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:39.677380 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.668352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.15.127.190:43345)
0924 07:16:39.668486 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.668491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.668505 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:39.668551 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.668554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.668609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.668702 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.669454 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.670301 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.671578 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.671916 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.672004 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.673664 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.673669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.673672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.673703 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.675466 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.675969 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.675974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.675976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.676069 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.676376 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.676382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.676384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.676597 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.676722 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.677061 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.677184 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:16:39.677395 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.666874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:57558)
0924 07:16:39.667301 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.667319 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.667444 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.668853 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.668859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.669150 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.669297 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.669309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.670478 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.670482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.671428 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.671439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.672205 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.672207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.672274 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.672955 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.672971 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.673463 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.675617 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.675624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.675626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.675858 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.676173 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.676178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.676180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.676277 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.676708 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.676712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.676830 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.676946 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.677185 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":92,"threads_started":1}
I20260924 07:16:39.678045 21996 tablet_copy_service.cc:140] P 21b39edfe646474880d6855985f97bc5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 87db3c4cfe024dadb0995f9bf4f44719 ({username='oryx', principal='oryx/127.15.127.188@KRBTEST.COM'} at 127.0.0.1:57558)
I20260924 07:16:39.678160 21996 tablet_copy_service.cc:161] P 21b39edfe646474880d6855985f97bc5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 87db3c4cfe024dadb0995f9bf4f44719 at {username='oryx', principal='oryx/127.15.127.188@KRBTEST.COM'} at 127.0.0.1:57558: session id = 87db3c4cfe024dadb0995f9bf4f44719-00000000000000000000000000000000
I20260924 07:16:39.679086 21996 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:16:39.680482 22544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:16:39.682688 22544 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: tablet copy: Starting download of 0 data blocks...
I20260924 07:16:39.682914 22544 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: tablet copy: Starting download of 1 WAL segments...
I20260924 07:16:39.684806 22544 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:16:39.687065 21996 tablet_copy_service.cc:342] P 21b39edfe646474880d6855985f97bc5: Request end of tablet copy session 87db3c4cfe024dadb0995f9bf4f44719-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.127.188@KRBTEST.COM'} at 127.0.0.1:57558
I20260924 07:16:39.687250 21996 tablet_copy_service.cc:433] P 21b39edfe646474880d6855985f97bc5: ending tablet copy session 87db3c4cfe024dadb0995f9bf4f44719-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 87db3c4cfe024dadb0995f9bf4f44719
I20260924 07:16:39.694023 22544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:39.695794 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.696064 22737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.696576 22544 server_base.cc:1034] running on GCE node
W20260924 07:16:39.697167 22735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.697402 22544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:39.697748 22544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:39.698889 22544 hybrid_clock.cc:648] HybridClock initialized: now 1790234199698887 us; error 23 us; skew 500 ppm
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:39.700371 22544 init.cc:377] Logged in from keytab as oryx/127.15.127.188@KRBTEST.COM (short username oryx)
I20260924 07:16:39.700865 22544 webserver.cc:460] Webserver started at http://127.15.127.188:46849/ using document root <none> and password file <none>
I20260924 07:16:39.701094 22544 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.701134 22544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.702450 22544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:39.703395 22744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.703632 22544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:39.703725 22544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
uuid: "87db3c4cfe024dadb0995f9bf4f44719"
format_stamp: "Formatted at 2026-09-24 07:16:39 on dist-test-slave-zx46"
server_key: "790abb9b849869060d79e10ae2ff01fd"
server_key_iv: "3b2ab079de2492b9a5c8481b0e31de08"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.703812 22544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.712649 22544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:39.714905 22544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:39.719801 22544 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.188:45171
I20260924 07:16:39.719805 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.188:45171 every 8 connection(s)
I20260924 07:16:39.720415 22544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
I20260924 07:16:39.722028 22797 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:16:39.722963 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Bootstrap starting.
I20260924 07:16:39.732259 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:16:39.732609 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: Bootstrap complete.
I20260924 07:16:39.733263 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } }
I20260924 07:16:39.733409 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 87db3c4cfe024dadb0995f9bf4f44719, State: Initialized, Role: LEARNER
I20260924 07:16:39.733526 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } }
I20260924 07:16:39.733880 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:39.734051 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:39.734062 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:16:39.737448 22811 catalog_manager.cc:1259] Loaded cluster ID: 06e2379f12f84b1dab79dbd49b03d188
I20260924 07:16:39.737533 22811 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: loading cluster ID for follower catalog manager: success
I20260924 07:16:39.739104 22811 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: acquiring CA information for follower catalog manager: success
I20260924 07:16:39.740252 22811 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:16:39.767232 22301 heartbeater.cc:503] Master 127.15.127.190:43345 was elected leader, sending a full tablet report...
W20260924 07:16:39.781127 22722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:39.781417 22722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:39.781461 22722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:39.785665 22722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:39.785734 22722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:39.785827 22722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:39.791246 22722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:37253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=45785
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:39.792441 22722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:39.793551 22722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:39.804812 22817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.804867 22722 server_base.cc:1034] running on GCE node
W20260924 07:16:39.804752 22819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:39.804730 22816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:39.805544 22722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:39.806257 22722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:39.807446 22722 hybrid_clock.cc:648] HybridClock initialized: now 1790234199807420 us; error 41 us; skew 500 ppm
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:39.810462 22722 init.cc:377] Logged in from keytab as oryx/127.15.127.129@KRBTEST.COM (short username oryx)
I20260924 07:16:39.811379 22722 webserver.cc:460] Webserver started at http://127.15.127.129:45785/ using document root <none> and password file <none>
I20260924 07:16:39.811967 22722 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:39.812076 22722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:39.816268 22722 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:39.818549 22826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.820051 22722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:16:39.820188 22722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "6c773f86301f403f83ff3271ace84f56"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "af94b9bdebda2e9f6a6e08efe2053132"
server_key_iv: "e171d245f011b459a00e65b4f85bcf57"
server_key_version: "encryptionkey@0"
I20260924 07:16:39.820833 22722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:39.844087 22722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:39.847666 22722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:39.847883 22722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:39.848536 22722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:39.849512 22722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:39.849560 22722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.849601 22722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:39.849647 22722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:39.859728 22722 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:37253
I20260924 07:16:39.859748 22939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:37253 every 8 connection(s)
I20260924 07:16:39.860822 22722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:39.865980 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22722
I20260924 07:16:39.866182 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22172
I20260924 07:16:39.873705 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:38543
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:43345
--webserver_port=33799
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
Sep 24 07:16:39 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234199, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.188@KRBTEST.COM
I20260924 07:16:39.879851 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.862697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:51055 (local address 127.15.127.190:43345)
0924 07:16:39.862853 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.862857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.864247 (+ 1390us) server_negotiation.cc:407] Connection header received
0924 07:16:39.865401 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.865405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.865465 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.865526 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.867862 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.868692 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.870583 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.870929 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.871003 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.875780 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.875786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.875793 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.875829 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.877575 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878183 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.878189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.878194 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.878285 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878595 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.878601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.878605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.878868 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.878955 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.879441 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.879588 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:16:39.879871 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.864871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:34741 (local address 127.15.127.188:45171)
0924 07:16:39.865162 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.865168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.865344 (+ 176us) server_negotiation.cc:407] Connection header received
0924 07:16:39.865554 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.865557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.865606 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.865669 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.868784 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.870018 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.871600 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.872062 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.872156 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.875790 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.875793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.875796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.875830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.877682 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878260 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.878265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.878267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.878376 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878686 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.878693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.878695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.878902 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.879003 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.879589 (+ 586us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.879716 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":78,"threads_started":1}
I20260924 07:16:39.880663 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.865119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.15.127.129:42453)
0924 07:16:39.865326 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.865341 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.865479 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.867967 (+ 2488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.867970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.868179 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.868296 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.868302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.869319 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.869321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.870447 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.870453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.871206 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.871210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.871639 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.872404 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.872437 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.874790 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.877311 (+ 2521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.877318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.877339 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.877732 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.878052 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.878060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.878275 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.878774 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.878781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.879127 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.879351 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.879726 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":226,"spinlock_wait_cycles":9088,"thread_start_us":87,"threads_started":1}
I20260924 07:16:39.881330 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.863154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:51055)
0924 07:16:39.864097 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.864140 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.865150 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.865943 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.865953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.866967 (+ 1014us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.867693 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.867717 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.868855 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.868858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.870442 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.870453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.871206 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.871210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.871637 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.872523 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.872541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.875601 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.877703 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.877708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.877713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.878063 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.878397 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.878408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.878499 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.879017 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.879022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.879152 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.880509 (+ 1357us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.880702 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":246,"spinlock_wait_cycles":24064,"thread_start_us":634,"threads_started":1}
I20260924 07:16:39.881551 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.864707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.15.127.129:34741)
0924 07:16:39.865265 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:39.865279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:39.865409 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:39.865943 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:39.865953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:39.868475 (+ 2522us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:39.868629 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.868639 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.870168 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.870174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.871452 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:39.871465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.872343 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.872348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:39.872509 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.873125 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:39.873142 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:39.875616 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:39.877804 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.877808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.877810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.878135 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.878479 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:39.878483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:39.878575 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:39.879043 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:39.879045 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:39.879130 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 07:16:39.879351 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 07:16:39.881351 (+ 2000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1822,"spinlock_wait_cycles":5248,"thread_start_us":106,"threads_started":1}
I20260924 07:16:39.882285 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:39.865474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:42453 (local address 127.15.127.189:44671)
0924 07:16:39.865574 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:39.865578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:39.865592 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:39.867690 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:39.867694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:39.867758 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:39.867851 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:39.868428 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.869185 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.870583 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:39.870925 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:39.871003 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:39.874992 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:39.874996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:39.875000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:39.875029 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:39.877180 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.877844 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.877848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.877850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.877940 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:39.878381 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:39.878385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:39.878386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:39.878635 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:39.878722 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:39.882005 (+ 3283us) server_negotiation.cc:299] Negotiation successful
0924 07:16:39.882140 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 07:16:39.882555 22941 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43345
I20260924 07:16:39.882840 22941 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:39.882875 22942 heartbeater.cc:347] Connected to a master server at 127.15.127.189:44671
I20260924 07:16:39.882965 22942 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:39.883459 22940 heartbeater.cc:347] Connected to a master server at 127.15.127.188:45171
I20260924 07:16:39.883589 22940 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:39.883735 22940 heartbeater.cc:511] Master 127.15.127.188:45171 requested a full tablet report, sending...
I20260924 07:16:39.883426 22942 heartbeater.cc:511] Master 127.15.127.189:44671 requested a full tablet report, sending...
I20260924 07:16:39.883430 22941 heartbeater.cc:511] Master 127.15.127.190:43345 requested a full tablet report, sending...
I20260924 07:16:39.885214 21976 ts_manager.cc:194] Re-registered known tserver with Master: 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253)
I20260924 07:16:39.885367 22758 ts_manager.cc:194] Registered new tserver with Master: 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253)
I20260924 07:16:39.885367 22578 ts_manager.cc:194] Registered new tserver with Master: 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253)
I20260924 07:16:39.886116 21976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:51055
I20260924 07:16:39.986493 22435 heartbeater.cc:503] Master 127.15.127.190:43345 was elected leader, sending a full tablet report...
W20260924 07:16:39.993364 22948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:39.993629 22948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:39.993673 22948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:39.997766 22948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:39.997831 22948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:39.997892 22948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:40.003305 22948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:38543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=33799
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:40.004504 22948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:40.005623 22948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:40.015688 22953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:40.015749 22956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:40.016031 22954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:40.021582 22948 server_base.cc:1034] running on GCE node
I20260924 07:16:40.022217 22948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:40.023141 22948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:40.026467 22948 hybrid_clock.cc:648] HybridClock initialized: now 1790234200026377 us; error 99 us; skew 500 ppm
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:40.029350 22948 init.cc:377] Logged in from keytab as oryx/127.15.127.130@KRBTEST.COM (short username oryx)
I20260924 07:16:40.030318 22948 webserver.cc:460] Webserver started at http://127.15.127.130:33799/ using document root <none> and password file <none>
I20260924 07:16:40.030915 22948 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:40.030989 22948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:40.035254 22948 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:40.037505 22963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:40.039098 22948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:40.039268 22948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "5c315fa90cb245a9b4cfe62af1e739bb"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "9cc4a68002777e39bf5ac291f44b37fa"
server_key_iv: "77907f4f3a0a5f335df2df0d68a24efc"
server_key_version: "encryptionkey@0"
I20260924 07:16:40.039901 22948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:40.060951 22948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:40.064119 22948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:40.064416 22948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:40.065186 22948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:40.066084 22948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:40.066161 22948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:40.066227 22948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:40.066277 22948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:40.076215 22948 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:38543
I20260924 07:16:40.076246 23076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:38543 every 8 connection(s)
I20260924 07:16:40.077383 22948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:40.083039 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 22948
I20260924 07:16:40.083222 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22306
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.188@KRBTEST.COM
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:40.090341 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:34001
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:43345
--webserver_port=43047
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
I20260924 07:16:40.096639 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.081137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:51089 (local address 127.15.127.188:45171)
0924 07:16:40.081263 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.081281 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.081421 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:16:40.081717 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.081721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.081781 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.081879 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.083975 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.084599 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.086290 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.086632 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.086702 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.091665 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.091669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.091671 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.091699 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.094056 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.094735 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.094739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.094742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.094838 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.095179 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.095186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.095189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.095444 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.095541 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.096212 (+ 671us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.096380 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:16:40.097384 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.079254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:43543 (local address 127.15.127.190:43345)
0924 07:16:40.079399 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.079403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.080636 (+ 1233us) server_negotiation.cc:407] Connection header received
0924 07:16:40.081740 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.081745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.081814 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.081878 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.084600 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.085413 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.086490 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.086827 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.086903 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.091672 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.091678 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.091683 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.091722 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.095581 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.096071 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.096074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.096076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.096172 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.096455 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.096458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.096460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.096677 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.096747 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.097134 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.097252 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:16:40.097927 23081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.079575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:43543)
0924 07:16:40.080461 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.080505 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.081480 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.082057 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.082068 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.082638 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:40.084449 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.084463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.085545 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.085549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.086378 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.086385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.087043 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.087045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.087115 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.087903 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:40.087932 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:40.091492 (+ 3560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:40.095707 (+ 4215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.095710 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.095712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.095950 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.096277 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.096279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.096281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.096357 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.096815 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:40.096818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:40.096895 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.097595 (+ 700us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.097727 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":194,"spinlock_wait_cycles":70144,"thread_start_us":540,"threads_started":1}
I20260924 07:16:40.098074 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.081400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.15.127.130:52517)
0924 07:16:40.081938 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.081951 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.082041 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.082285 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.082290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.082613 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:40.085606 (+ 2993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.085621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.086814 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.086820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.087910 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.087922 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.088784 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.088790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.088911 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.089609 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:40.089629 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:40.091443 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:40.093762 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.093772 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.093796 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.094194 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.094538 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.094542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.094545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.094775 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.095467 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:40.095475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:40.095893 (+ 418us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.096154 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.096565 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":132,"spinlock_wait_cycles":11648,"thread_start_us":82,"threads_started":1}
I20260924 07:16:40.097945 23082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.080997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.15.127.130:51089)
0924 07:16:40.081339 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.081353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.081497 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.082057 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.082066 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.082593 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:40.083801 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.083825 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.084774 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.084777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.086159 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.086168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.086904 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.086908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.087109 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.087985 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:40.087998 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:40.091443 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:40.094318 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.094324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.094329 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.094605 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.094959 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.094964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.094968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.095066 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.095619 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:40.095625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:40.095907 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.097196 (+ 1289us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.097379 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":81,"spinlock_wait_cycles":158848,"thread_start_us":118,"threads_started":1}
I20260924 07:16:40.098906 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.081686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:52517 (local address 127.15.127.189:44671)
0924 07:16:40.081809 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.081813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.082015 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:16:40.082116 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.082119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.082165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.082251 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.085756 (+ 3505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.086666 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.088051 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.088500 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.088592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.091660 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.091664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.091667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.091697 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.093628 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.094317 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.094321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.094322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.094417 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.094899 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.094902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.094904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.095277 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.095400 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.098607 (+ 3207us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.098750 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:16:40.099326 23078 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43345
I20260924 07:16:40.099472 23079 heartbeater.cc:347] Connected to a master server at 127.15.127.189:44671
I20260924 07:16:40.099599 23079 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:40.099608 23078 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:40.100227 23077 heartbeater.cc:347] Connected to a master server at 127.15.127.188:45171
I20260924 07:16:40.100255 23079 heartbeater.cc:511] Master 127.15.127.189:44671 requested a full tablet report, sending...
I20260924 07:16:40.100317 23077 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:40.100257 23078 heartbeater.cc:511] Master 127.15.127.190:43345 requested a full tablet report, sending...
I20260924 07:16:40.100456 23077 heartbeater.cc:511] Master 127.15.127.188:45171 requested a full tablet report, sending...
I20260924 07:16:40.101450 21976 ts_manager.cc:194] Re-registered known tserver with Master: 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543)
I20260924 07:16:40.101480 22578 ts_manager.cc:194] Registered new tserver with Master: 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543)
I20260924 07:16:40.101513 22758 ts_manager.cc:194] Registered new tserver with Master: 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543)
I20260924 07:16:40.102240 21976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:43543
I20260924 07:16:40.119378 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.115593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43498)
0924 07:16:40.115773 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.115791 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.115883 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.116133 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.116136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.116170 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:40.116258 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.116262 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.117293 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.117295 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.118025 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.118033 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.118967 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.118970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.119075 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.119105 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.119196 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.119262 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260924 07:16:40.119426 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.115733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.15.127.188:45171)
0924 07:16:40.115877 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.115882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.115900 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:16:40.115951 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.115954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.116018 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.116119 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:40.116365 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.117193 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.118143 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.118819 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.118947 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.119082 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.119226 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.119294 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:16:40.120275 22771 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 07:16:40.120707 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b39edfe646474880d6855985f97bc5. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:40.120854 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: This master's current role is: LEARNER
W20260924 07:16:40.211040 23084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:40.211364 23084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:40.211449 23084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:40.215654 23084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:40.215723 23084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:40.215787 23084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:40.221262 23084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:34001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=43047
--tserver_master_addrs=127.15.127.190:43345,127.15.127.189:44671,127.15.127.188:45171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:40.222337 23084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:40.223333 23084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:40.233390 23090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:40.233402 23089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:40.233462 23092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:40.233544 23084 server_base.cc:1034] running on GCE node
I20260924 07:16:40.234097 23084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:40.234707 23084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:40.235921 23084 hybrid_clock.cc:648] HybridClock initialized: now 1790234200235895 us; error 29 us; skew 500 ppm
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:40.238638 23084 init.cc:377] Logged in from keytab as oryx/127.15.127.131@KRBTEST.COM (short username oryx)
I20260924 07:16:40.239653 23084 webserver.cc:460] Webserver started at http://127.15.127.131:43047/ using document root <none> and password file <none>
I20260924 07:16:40.240288 23084 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:40.240378 23084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:40.245210 23084 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:40.247571 23099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:40.249111 23084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:40.249223 23084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "930a85194c5f41dbb7fdad053539fbce"
format_stamp: "Formatted at 2026-09-24 07:16:38 on dist-test-slave-zx46"
server_key: "36ea3fd5ede43e5d073a219e0f9c2a19"
server_key_iv: "2c885c8f1b74b4a31ab3bc3ad81ed2f6"
server_key_version: "encryptionkey@0"
I20260924 07:16:40.249917 23084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:40.286587 23084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:40.289642 23084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:40.289800 23084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:40.290335 23084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:40.291148 23084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:40.291194 23084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:40.291234 23084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:40.291250 23084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:40.300527 23084 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:34001
I20260924 07:16:40.300545 23212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:34001 every 8 connection(s)
I20260924 07:16:40.301661 23084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:40.309082 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 23084
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.188@KRBTEST.COM
Sep 24 07:16:40 dist-test-slave-zx46 krb5kdc[21931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234200, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.189@KRBTEST.COM
I20260924 07:16:40.319380 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.303488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:34543 (local address 127.15.127.189:44671)
0924 07:16:40.303635 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.303638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.304997 (+ 1359us) server_negotiation.cc:407] Connection header received
0924 07:16:40.306275 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.306279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.306334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.306388 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.307854 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.308688 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310781 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.311132 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.311202 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.314646 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.314652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.314654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.314684 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.316344 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.317091 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.317094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.317096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.317182 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.317587 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.317590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.317592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.317827 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.317933 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.319014 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.319149 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:16:40.320225 23218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.305354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.15.127.131:33817)
0924 07:16:40.305638 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.305652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.306024 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.306645 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.306653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.307121 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:40.307709 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.307715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.308842 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.308845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.310093 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.310107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.310794 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.310892 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.311603 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:40.311622 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:40.314125 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:40.316342 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.316349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.316367 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.316722 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.317316 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.317321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.317325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.317504 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.318049 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:40.318056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:40.318426 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.318642 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.319066 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":261,"spinlock_wait_cycles":63232,"thread_start_us":86,"threads_started":1}
I20260924 07:16:40.320518 23217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.303961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.15.127.131:34543)
0924 07:16:40.304840 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.304885 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.306024 (+ 1139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.306646 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.306660 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.307122 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:40.307707 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.307715 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.309132 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.309135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.310655 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.310662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.311397 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.311403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.311489 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.311955 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:40.311969 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:40.314470 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:40.316465 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.316468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.316470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.316961 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.317290 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.317293 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.317295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.317486 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.318122 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:40.318125 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:40.318425 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.318643 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.319646 (+ 1003us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":1490,"spinlock_wait_cycles":11904,"thread_start_us":539,"threads_started":1}
I20260924 07:16:40.321583 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.305496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:33817 (local address 127.15.127.188:45171)
0924 07:16:40.305628 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.305635 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.305739 (+ 104us) server_negotiation.cc:407] Connection header received
0924 07:16:40.306275 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.306279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.306334 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.306388 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.307854 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.308688 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310260 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.310547 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310629 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.314313 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.314317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.314320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.314350 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.316211 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.317098 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.317104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.317106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.317202 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.317621 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.317628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.317630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.317891 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.317989 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.321286 (+ 3297us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.321405 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:16:40.321748 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.306123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:41251 (local address 127.15.127.190:43345)
0924 07:16:40.306247 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.306251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.306264 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:40.306311 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.306314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.306366 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.306458 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.307856 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.308464 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310096 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.310408 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310526 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.316663 (+ 6137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.316666 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.316669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.316698 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.318465 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.320074 (+ 1609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.320078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.320079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.320181 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.320484 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.320487 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.320489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.320729 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.320817 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.321192 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.321314 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:40.321955 23215 heartbeater.cc:347] Connected to a master server at 127.15.127.189:44671
I20260924 07:16:40.322271 23215 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:40.322453 23213 heartbeater.cc:347] Connected to a master server at 127.15.127.188:45171
I20260924 07:16:40.322541 23213 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:40.322913 23215 heartbeater.cc:511] Master 127.15.127.189:44671 requested a full tablet report, sending...
I20260924 07:16:40.323549 23213 heartbeater.cc:511] Master 127.15.127.188:45171 requested a full tablet report, sending...
I20260924 07:16:40.322260 23219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.305736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:41251)
0924 07:16:40.305961 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.305978 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.306073 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.306644 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.306651 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.307118 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:40.307676 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.307691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.308609 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.308613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.309932 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.309943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.310566 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.310570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.310782 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.311453 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:40.311476 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:40.315555 (+ 4079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:40.319714 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.319717 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.319720 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.319954 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.320300 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:40.320303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:40.320305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:40.320383 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:40.320900 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:40.320906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:40.320987 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.321943 (+ 956us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.322081 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":18,"spinlock_wait_cycles":20864,"thread_start_us":98,"threads_started":1}
I20260924 07:16:40.324213 22578 ts_manager.cc:194] Registered new tserver with Master: 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001)
I20260924 07:16:40.324257 22758 ts_manager.cc:194] Registered new tserver with Master: 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001)
I20260924 07:16:40.324374 23214 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43345
I20260924 07:16:40.324451 23214 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:40.324587 23214 heartbeater.cc:511] Master 127.15.127.190:43345 requested a full tablet report, sending...
I20260924 07:16:40.325547 21976 ts_manager.cc:194] Re-registered known tserver with Master: 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001)
I20260924 07:16:40.326550 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.313351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.15.127.188:45171)
0924 07:16:40.313610 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.313616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.314089 (+ 473us) server_negotiation.cc:407] Connection header received
0924 07:16:40.314146 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.314150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.314206 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.314287 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.315890 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.316689 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.317869 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.318204 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.318295 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.321120 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.321124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.321127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.321151 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.323696 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.324937 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.324942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.324944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.325059 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.325365 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.325369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.325370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.325590 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.325774 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.326210 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.326348 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":82,"threads_started":1}
I20260924 07:16:40.326756 21976 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:41251
I20260924 07:16:40.331364 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.312185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.15.127.190:43345)
0924 07:16:40.313313 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.313317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.313331 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:40.313370 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.313373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.313430 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.314511 (+ 1081us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.315880 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.316712 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.325360 (+ 8648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.325680 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.325927 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.327408 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.327412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.327414 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.327439 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.329495 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.329974 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.329978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.329980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.330067 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.330485 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.330491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.330492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.330754 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.330869 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.331111 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.331214 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1045,"thread_start_us":74,"threads_started":1}
I20260924 07:16:40.333213 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.312996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57728 (local address 127.15.127.189:44671)
0924 07:16:40.313952 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.313963 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.320216 (+ 6253us) server_negotiation.cc:407] Connection header received
0924 07:16:40.320266 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.320271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.320318 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.320411 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.322814 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.323810 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.326352 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.327702 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.327798 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.329239 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.329246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.329249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.329279 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.331275 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.331857 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.331860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.331861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.331938 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.332236 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.332239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.332241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.332427 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.332581 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.332821 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.332967 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":841,"thread_start_us":72,"threads_started":1}
I20260924 07:16:40.334560 21976 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:40.337774 21976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:40.340205 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: NON_VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: true } } }
I20260924 07:16:40.340361 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:40.344380 23233 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: attempting to promote NON_VOTER 87db3c4cfe024dadb0995f9bf4f44719 to VOTER
I20260924 07:16:40.344795 23233 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:40.345973 22587 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Refusing update from remote peer 21b39edfe646474880d6855985f97bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 07:16:40.346056 22771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 LEARNER]: Refusing update from remote peer 21b39edfe646474880d6855985f97bc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 07:16:40.346534 23232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 07:16:40.346665 22536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 07:16:40.348711 23231 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.349176 22587 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.349777 22771 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.350276 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.350380 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:40.350598 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.350694 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:40.350795 23231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.350898 23231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:40.351832 23241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.351961 23241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:40.352039 22800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21b39edfe646474880d6855985f97bc5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:40.352139 22800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:40.357791 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.352439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:34001 (local address 127.0.0.1:41750)
0924 07:16:40.352599 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.352616 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.352712 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.354070 (+ 1358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.354076 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.354100 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:40.354226 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.354235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.355577 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.355584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.356335 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.356343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.357420 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.357424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.357510 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.357536 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.357581 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.357647 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 07:16:40.358528 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.353065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41750 (local address 127.15.127.131:34001)
0924 07:16:40.353525 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.353541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.353568 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:16:40.353652 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.353657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.353948 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.354085 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:40.354367 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.355468 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.356457 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.357275 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.358088 (+ 813us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.358243 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.358297 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.358388 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":96,"threads_started":1}
I20260924 07:16:40.359426 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.352658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37253 (local address 127.0.0.1:40046)
0924 07:16:40.353165 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.353179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.353267 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.354960 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.354963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.354982 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:40.355075 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.355083 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.356472 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.356476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.357731 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.357739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.358847 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.358850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.359052 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.359114 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.359182 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.359292 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":159,"thread_start_us":131,"threads_started":1}
I20260924 07:16:40.359536 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.353606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.15.127.130:38543)
0924 07:16:40.353984 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.353996 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.354482 (+ 486us) server_negotiation.cc:407] Connection header received
0924 07:16:40.354563 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.354568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.354781 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.354923 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:40.355304 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.356374 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.357818 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.358473 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.358616 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.358747 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.359288 (+ 541us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.359390 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":200,"threads_started":1}
I20260924 07:16:40.359822 23245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.353200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38543 (local address 127.0.0.1:53232)
0924 07:16:40.354221 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.354234 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.354325 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.355069 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.355071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.355089 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:40.355187 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.355192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.356490 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.356493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.357695 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.357704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.358847 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.358850 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.359056 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.359105 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.359182 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.359731 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"mutex_wait_us":552,"thread_start_us":94,"threads_started":1}
I20260924 07:16:40.361737 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.353105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.15.127.129:37253)
0924 07:16:40.354397 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.354413 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.354436 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:16:40.354517 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.354523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.354742 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.354991 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:40.355171 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.356332 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.357859 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.358691 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.358854 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.358963 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.359225 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.359317 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1185,"thread_start_us":98,"threads_started":1}
I20260924 07:16:40.361497 23147 tablet_service.cc:1467] Processing CreateTablet for tablet c4a990c2567147649f347c25de16ec69 (DEFAULT_TABLE table=test-table [id=e57230b8fdb549a7b82528cb9e5452d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:40.362685 23147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a990c2567147649f347c25de16ec69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:40.363785 23011 tablet_service.cc:1467] Processing CreateTablet for tablet c4a990c2567147649f347c25de16ec69 (DEFAULT_TABLE table=test-table [id=e57230b8fdb549a7b82528cb9e5452d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:40.364044 22874 tablet_service.cc:1467] Processing CreateTablet for tablet c4a990c2567147649f347c25de16ec69 (DEFAULT_TABLE table=test-table [id=e57230b8fdb549a7b82528cb9e5452d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:40.364930 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a990c2567147649f347c25de16ec69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:40.365101 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a990c2567147649f347c25de16ec69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:40.371452 23247 tablet_bootstrap.cc:492] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Bootstrap starting.
I20260924 07:16:40.372314 23248 tablet_bootstrap.cc:492] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Bootstrap starting.
I20260924 07:16:40.373260 23249 tablet_bootstrap.cc:492] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Bootstrap starting.
I20260924 07:16:40.374431 23248 tablet_bootstrap.cc:654] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:40.374437 23247 tablet_bootstrap.cc:654] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:40.375331 23248 log.cc:824] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:40.375464 23249 tablet_bootstrap.cc:654] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:40.375852 23247 log.cc:824] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:40.376224 23249 log.cc:824] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:40.377391 23248 tablet_bootstrap.cc:492] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: No bootstrap required, opened a new log
I20260924 07:16:40.377602 23248 ts_tablet_manager.cc:1397] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:16:40.377766 23247 tablet_bootstrap.cc:492] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: No bootstrap required, opened a new log
I20260924 07:16:40.377951 23247 ts_tablet_manager.cc:1397] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:16:40.378568 23249 tablet_bootstrap.cc:492] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: No bootstrap required, opened a new log
I20260924 07:16:40.378750 23249 ts_tablet_manager.cc:1397] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 07:16:40.381906 23247 raft_consensus.cc:348] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.381906 23248 raft_consensus.cc:348] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.382215 23247 raft_consensus.cc:374] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:40.382215 23248 raft_consensus.cc:374] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:40.382272 23248 raft_consensus.cc:729] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930a85194c5f41dbb7fdad053539fbce, State: Initialized, Role: FOLLOWER
I20260924 07:16:40.382268 23247 raft_consensus.cc:729] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c773f86301f403f83ff3271ace84f56, State: Initialized, Role: FOLLOWER
I20260924 07:16:40.382788 23248 consensus_queue.cc:260] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.383169 23247 consensus_queue.cc:260] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.383870 22941 heartbeater.cc:503] Master 127.15.127.190:43345 was elected leader, sending a full tablet report...
I20260924 07:16:40.384109 23248 ts_tablet_manager.cc:1428] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:16:40.384194 23247 ts_tablet_manager.cc:1428] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:16:40.383548 23249 raft_consensus.cc:348] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.384528 23249 raft_consensus.cc:374] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:40.384583 23249 raft_consensus.cc:729] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c315fa90cb245a9b4cfe62af1e739bb, State: Initialized, Role: FOLLOWER
I20260924 07:16:40.385087 23249 consensus_queue.cc:260] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.385980 23249 ts_tablet_manager.cc:1428] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 07:16:40.386505 23214 heartbeater.cc:503] Master 127.15.127.190:43345 was elected leader, sending a full tablet report...
I20260924 07:16:40.387269 23078 heartbeater.cc:503] Master 127.15.127.190:43345 was elected leader, sending a full tablet report...
I20260924 07:16:40.521504 23254 raft_consensus.cc:482] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:40.521754 23254 raft_consensus.cc:504] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.523058 23254 leader_election.cc:288] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001), 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543)
I20260924 07:16:40.527236 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.523439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:34339 (local address 127.15.127.131:34001)
0924 07:16:40.523622 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.523627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.523743 (+ 116us) server_negotiation.cc:407] Connection header received
0924 07:16:40.524011 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.524013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.524074 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.524145 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:40.524440 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.525137 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.526305 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.526707 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.526857 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.526913 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.527050 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.527114 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 07:16:40.527258 23256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.523292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:34001 (local address 127.15.127.129:34339)
0924 07:16:40.523649 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.523666 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.523870 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.524174 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.524177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.524193 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:40.524298 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.524304 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.525253 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.525259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.526176 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.526185 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.526831 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.526835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.526913 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.526945 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.527023 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.527098 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":37,"thread_start_us":80,"threads_started":1}
I20260924 07:16:40.527362 23246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.523541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:40763 (local address 127.15.127.130:38543)
0924 07:16:40.523706 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.523713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.523870 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:16:40.524064 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.524066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.524107 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.524191 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:40.524483 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.525189 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.526399 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.526800 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.526923 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.526972 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.527154 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.527225 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:40.527786 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.523432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38543 (local address 127.15.127.129:40763)
0924 07:16:40.523783 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:40.523799 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:40.523923 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:40.524200 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:40.524203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:40.524219 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:40.524334 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.524339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.525300 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.525306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.526275 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:40.526284 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.526918 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.526922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:40.527009 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.527030 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:16:40.527138 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:16:40.527205 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":91,"threads_started":1}
I20260924 07:16:40.528134 23167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a990c2567147649f347c25de16ec69" candidate_uuid: "6c773f86301f403f83ff3271ace84f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930a85194c5f41dbb7fdad053539fbce" is_pre_election: true
I20260924 07:16:40.528436 23031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a990c2567147649f347c25de16ec69" candidate_uuid: "6c773f86301f403f83ff3271ace84f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" is_pre_election: true
I20260924 07:16:40.528467 23167 raft_consensus.cc:2454] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c773f86301f403f83ff3271ace84f56 in term 0.
I20260924 07:16:40.528724 23031 raft_consensus.cc:2454] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c773f86301f403f83ff3271ace84f56 in term 0.
I20260924 07:16:40.528983 22830 leader_election.cc:302] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c773f86301f403f83ff3271ace84f56, 930a85194c5f41dbb7fdad053539fbce; no voters:
I20260924 07:16:40.529306 23254 raft_consensus.cc:2781] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:40.529386 23254 raft_consensus.cc:482] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:40.529417 23254 raft_consensus.cc:3037] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:40.530499 23254 raft_consensus.cc:504] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.530882 23254 leader_election.cc:288] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [CANDIDATE]: Term 1 election: Requested vote from peers 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001), 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543)
I20260924 07:16:40.531281 23167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a990c2567147649f347c25de16ec69" candidate_uuid: "6c773f86301f403f83ff3271ace84f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930a85194c5f41dbb7fdad053539fbce"
I20260924 07:16:40.531270 23031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4a990c2567147649f347c25de16ec69" candidate_uuid: "6c773f86301f403f83ff3271ace84f56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c315fa90cb245a9b4cfe62af1e739bb"
I20260924 07:16:40.531404 23031 raft_consensus.cc:3037] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:40.531404 23167 raft_consensus.cc:3037] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:40.532701 23031 raft_consensus.cc:2454] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c773f86301f403f83ff3271ace84f56 in term 1.
I20260924 07:16:40.532701 23167 raft_consensus.cc:2454] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c773f86301f403f83ff3271ace84f56 in term 1.
I20260924 07:16:40.533025 22830 leader_election.cc:302] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c773f86301f403f83ff3271ace84f56, 930a85194c5f41dbb7fdad053539fbce; no voters:
I20260924 07:16:40.533201 23254 raft_consensus.cc:2781] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:40.533474 23254 raft_consensus.cc:686] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 1 LEADER]: Becoming Leader. State: Replica: 6c773f86301f403f83ff3271ace84f56, State: Running, Role: LEADER
I20260924 07:16:40.533839 23254 consensus_queue.cc:237] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } }
I20260924 07:16:40.537586 21976 catalog_manager.cc:5495] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c773f86301f403f83ff3271ace84f56 (127.15.127.129). New cstate: current_term: 1 leader_uuid: "6c773f86301f403f83ff3271ace84f56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } health_report { overall_health: UNKNOWN } } }
W20260924 07:16:40.553131 23216 tablet.cc:2369] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:40.578967 23080 tablet.cc:2369] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:40.596006 23244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.592635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.15.127.129:37253)
0924 07:16:40.592779 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.592783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.592801 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:16:40.592901 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.592904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.592975 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.593123 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:40.593300 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.593792 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.594578 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.594786 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.594862 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.595117 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:40.595627 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:40.595726 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.595774 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.595826 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:16:40.602329 23031 raft_consensus.cc:1261] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Refusing update from remote peer 6c773f86301f403f83ff3271ace84f56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:40.602336 23167 raft_consensus.cc:1261] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Refusing update from remote peer 6c773f86301f403f83ff3271ace84f56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:40.603142 23258 consensus_queue.cc:1035] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:40.603339 23254 consensus_queue.cc:1035] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:40.614161 23263 mvcc.cc:205] Tried to move back new op lower bound from 7332799285659369472 to 7332799285389750272. Current Snapshot: MvccSnapshot[applied={T|T < 7332799285659369472}]
I20260924 07:16:40.622069 23262 mvcc.cc:205] Tried to move back new op lower bound from 7332799285659369472 to 7332799285389750272. Current Snapshot: MvccSnapshot[applied={T|T < 7332799285659369472 or (T in {7332799285659369472})}]
I20260924 07:16:40.623229 21975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:40.623471 21975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:40.628160 21975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21b39edfe646474880d6855985f97bc5: Sending DeleteTablet for 3 replicas of tablet c4a990c2567147649f347c25de16ec69
I20260924 07:16:40.628933 22874 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a990c2567147649f347c25de16ec69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:40 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:40046
I20260924 07:16:40.629209 23147 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a990c2567147649f347c25de16ec69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:40 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:41750
I20260924 07:16:40.629271 23011 tablet_service.cc:1514] Processing DeleteTablet for tablet c4a990c2567147649f347c25de16ec69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:40 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:53232
I20260924 07:16:40.629973 23271 tablet_replica.cc:330] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: stopping tablet replica
I20260924 07:16:40.629973 23272 tablet_replica.cc:330] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: stopping tablet replica
I20260924 07:16:40.630326 23272 raft_consensus.cc:2229] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:40.630326 23271 raft_consensus.cc:2229] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:40.630649 23272 raft_consensus.cc:2258] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:40.630658 23271 raft_consensus.cc:2258] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:40.632223 23271 ts_tablet_manager.cc:1905] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:40.632223 23272 ts_tablet_manager.cc:1905] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:40.633406 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 21944
I20260924 07:16:40.635309 23272 ts_tablet_manager.cc:1918] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:40.635421 23272 log.cc:1196] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/c4a990c2567147649f347c25de16ec69
I20260924 07:16:40.635496 23271 ts_tablet_manager.cc:1918] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:40.635582 23271 log.cc:1196] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/c4a990c2567147649f347c25de16ec69
I20260924 07:16:40.635819 23272 ts_tablet_manager.cc:1939] T c4a990c2567147649f347c25de16ec69 P 5c315fa90cb245a9b4cfe62af1e739bb: Deleting consensus metadata
I20260924 07:16:40.635896 23271 ts_tablet_manager.cc:1939] T c4a990c2567147649f347c25de16ec69 P 6c773f86301f403f83ff3271ace84f56: Deleting consensus metadata
I20260924 07:16:40.641697 23273 tablet_replica.cc:330] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: stopping tablet replica
W20260924 07:16:40.641726 22966 connection.cc:535] server connection from 127.0.0.1:53232 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:16:40.642136 23273 raft_consensus.cc:2229] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 07:16:40.642333 22827 connection.cc:535] server connection from 127.0.0.1:40046 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:16:40.642565 23273 raft_consensus.cc:2258] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:40.644484 23273 ts_tablet_manager.cc:1905] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:40.647619 23273 ts_tablet_manager.cc:1918] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:40.647742 23273 log.cc:1196] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/c4a990c2567147649f347c25de16ec69
I20260924 07:16:40.648129 23273 ts_tablet_manager.cc:1939] T c4a990c2567147649f347c25de16ec69 P 930a85194c5f41dbb7fdad053539fbce: Deleting consensus metadata
W20260924 07:16:40.649562 23103 connection.cc:441] server connection from 127.0.0.1:41750 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41750 (request call id 1) could send its response
I20260924 07:16:40.651307 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.631597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.15.127.189:44671)
0924 07:16:40.631848 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.631853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.641424 (+ 9571us) server_negotiation.cc:407] Connection header received
0924 07:16:40.641616 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.641619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.641680 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.641744 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.643013 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.643811 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.645230 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.645556 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.645636 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.647647 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.647650 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.647653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.647684 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.649506 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.650026 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.650030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.650032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.650124 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.650414 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.650417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.650419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.650604 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.650750 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.651014 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.651136 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260924 07:16:40.659044 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.646764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.15.127.189:44671)
0924 07:16:40.646875 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.646879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.648371 (+ 1492us) server_negotiation.cc:407] Connection header received
0924 07:16:40.648531 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.648534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.648583 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.648633 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.650837 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.651562 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.652608 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.652969 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.653078 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.655106 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.655111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.655114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.655138 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.656972 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.657520 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.657525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.657528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.657621 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.657961 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.657966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.657970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.658216 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.658327 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.658692 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.658826 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:16:40.659241 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.647352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.15.127.188:45171)
0924 07:16:40.647458 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.647462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.649921 (+ 2459us) server_negotiation.cc:407] Connection header received
0924 07:16:40.650318 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.650322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.650377 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.650428 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.651218 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.651980 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.653598 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.653938 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.654018 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.655677 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.655682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.655685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.655712 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.657422 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.657951 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.657954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.657956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.658048 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.658338 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.658342 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.658343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.658541 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.658639 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.658962 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.659085 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:16:40.661844 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:40.631732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.15.127.188:45171)
0924 07:16:40.631841 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:40.631847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:40.652690 (+ 20843us) server_negotiation.cc:407] Connection header received
0924 07:16:40.652852 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:40.652855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:40.652904 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:40.652952 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:40.654176 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.655225 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.656465 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:40.656836 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:40.656919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:40.658575 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:40.658581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:40.658584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:40.658616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:40.660252 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.660688 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.660691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.660693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.660776 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:40.661069 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:40.661072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:40.661074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:40.661264 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:40.661382 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:40.661585 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:16:40.661698 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:16:41.619062 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:41.618562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:42387)
0924 07:16:41.618822 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:41.618934 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":71,"threads_started":1}
W20260924 07:16:41.619418 22941 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:43345 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:41.619553 23283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:41.619121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:58053)
0924 07:16:41.619357 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:41.619448 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":81,"threads_started":1}
W20260924 07:16:41.619866 23214 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:43345 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:41.624833 23284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:41.624397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:43977)
0924 07:16:41.624618 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:41.624726 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
W20260924 07:16:41.625129 23078 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:43345 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:42.128825 23285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 21b39edfe646474880d6855985f97bc5)
I20260924 07:16:42.128966 23286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 21b39edfe646474880d6855985f97bc5)
I20260924 07:16:42.129089 23285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:42.129194 23286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:42.130084 23285 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345), 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188:45171)
I20260924 07:16:42.130115 23286 leader_election.cc:288] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345), 2f5f661bd8dc40e0972223ed165e9d4e (127.15.127.189:44671)
I20260924 07:16:42.130714 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.130270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58058)
0924 07:16:42.130494 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.130575 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260924 07:16:42.130879 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.130382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58066)
0924 07:16:42.130588 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.130661 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260924 07:16:42.131102 22748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
W20260924 07:16:42.131414 22565 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:42.131997 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.131725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58078)
0924 07:16:42.131852 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.131899 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 07:16:42.132089 22748 leader_election.cc:334] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:42.132467 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.132139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58092)
0924 07:16:42.132360 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.132406 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
W20260924 07:16:42.132558 22565 leader_election.cc:334] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:42.135475 23291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.130854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.15.127.189:44671)
0924 07:16:42.131348 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:42.131352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:42.131364 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:42.131401 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:42.131404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:42.131451 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:42.131830 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:42.131920 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.132632 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.134166 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.134855 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.134989 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:42.135106 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:42.135259 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:16:42.135339 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":428,"thread_start_us":65,"threads_started":1}
I20260924 07:16:42.135514 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.130675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:45171 (local address 127.0.0.1:43542)
0924 07:16:42.131049 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.131063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:42.131154 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:42.131452 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:42.131456 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:42.131477 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:42.131601 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:42.131611 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.132712 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.132718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:42.134054 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:42.134064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.135012 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.135015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:42.135128 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:42.135160 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 07:16:42.135235 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:16:42.135352 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260924 07:16:42.135531 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.131012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.15.127.188:45171)
0924 07:16:42.131203 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:42.131207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:42.131220 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:42.131262 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:42.131265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:42.131316 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:42.131730 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:42.131826 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.132569 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.134302 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.134919 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.135018 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:42.135075 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:42.135259 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 07:16:42.135388 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260924 07:16:42.135921 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.130587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:44671 (local address 127.0.0.1:57760)
0924 07:16:42.130852 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.130865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:42.130973 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:42.131574 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:42.131578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:42.131600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:42.131723 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:42.131733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.132816 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.132819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:42.134054 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:42.134064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:42.134992 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:42.134995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:42.135092 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:42.135117 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:16:42.135228 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:16:42.135414 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":61,"thread_start_us":65,"threads_started":1}
I20260924 07:16:42.135934 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87db3c4cfe024dadb0995f9bf4f44719" is_pre_election: true
I20260924 07:16:42.136179 22771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f5f661bd8dc40e0972223ed165e9d4e in term 1.
I20260924 07:16:42.136353 22587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87db3c4cfe024dadb0995f9bf4f44719" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" is_pre_election: true
I20260924 07:16:42.136538 22587 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87db3c4cfe024dadb0995f9bf4f44719 in term 1.
I20260924 07:16:42.136539 22565 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f5f661bd8dc40e0972223ed165e9d4e, 87db3c4cfe024dadb0995f9bf4f44719; no voters: 21b39edfe646474880d6855985f97bc5
I20260924 07:16:42.136807 23285 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:16:42.136873 23285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 21b39edfe646474880d6855985f97bc5)
I20260924 07:16:42.136924 23285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:16:42.136888 22748 leader_election.cc:302] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f5f661bd8dc40e0972223ed165e9d4e, 87db3c4cfe024dadb0995f9bf4f44719; no voters: 21b39edfe646474880d6855985f97bc5
I20260924 07:16:42.137120 23286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:16:42.137204 23286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 21b39edfe646474880d6855985f97bc5)
I20260924 07:16:42.137267 23286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:16:42.138031 23285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:42.138286 23286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:42.138417 23285 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 2 election: Requested vote from peers 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345), 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188:45171)
I20260924 07:16:42.138643 23286 leader_election.cc:288] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [CANDIDATE]: Term 2 election: Requested vote from peers 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345), 2f5f661bd8dc40e0972223ed165e9d4e (127.15.127.189:44671)
I20260924 07:16:42.138805 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.138496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58104)
0924 07:16:42.138597 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.138677 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:16:42.139096 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87db3c4cfe024dadb0995f9bf4f44719"
I20260924 07:16:42.139258 22771 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f5f661bd8dc40e0972223ed165e9d4e in current term 2: Already voted for candidate 87db3c4cfe024dadb0995f9bf4f44719 in this term.
I20260924 07:16:42.139243 22587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87db3c4cfe024dadb0995f9bf4f44719" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2f5f661bd8dc40e0972223ed165e9d4e"
I20260924 07:16:42.139374 22587 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 87db3c4cfe024dadb0995f9bf4f44719 in current term 2: Already voted for candidate 2f5f661bd8dc40e0972223ed165e9d4e in this term.
I20260924 07:16:42.139564 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.139234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58120)
0924 07:16:42.139445 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.139502 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:16:42.139582 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.138878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58116)
0924 07:16:42.139461 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.139517 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527}
W20260924 07:16:42.139729 22565 leader_election.cc:334] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:42.139801 22565 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f5f661bd8dc40e0972223ed165e9d4e; no voters: 21b39edfe646474880d6855985f97bc5, 87db3c4cfe024dadb0995f9bf4f44719
I20260924 07:16:42.140019 23285 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 07:16:42.140241 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.139999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58130)
0924 07:16:42.140138 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.140186 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:16:42.140339 22748 leader_election.cc:334] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:42.140399 22748 leader_election.cc:302] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 87db3c4cfe024dadb0995f9bf4f44719; no voters: 21b39edfe646474880d6855985f97bc5, 2f5f661bd8dc40e0972223ed165e9d4e
I20260924 07:16:42.140522 23286 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 07:16:42.620471 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.620068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:55411)
0924 07:16:42.620323 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.620381 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":104,"threads_started":1}
I20260924 07:16:42.620777 23296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.620390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:41127)
0924 07:16:42.620622 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.620682 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260924 07:16:42.626020 23297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:42.625627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:46193)
0924 07:16:42.625857 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:42.625922 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260924 07:16:43.621651 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.621184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:35143)
0924 07:16:43.621427 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.621509 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260924 07:16:43.621817 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.621426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:55007)
0924 07:16:43.621656 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.621716 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
I20260924 07:16:43.627154 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.626633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:46001)
0924 07:16:43.627012 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.627066 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":78,"threads_started":1}
I20260924 07:16:43.785977 23301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:43.786201 23301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:43.786659 23301 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345), 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188:45171)
I20260924 07:16:43.787389 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.786936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58290)
0924 07:16:43.787224 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.787305 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":101,"threads_started":1}
I20260924 07:16:43.787598 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87db3c4cfe024dadb0995f9bf4f44719" is_pre_election: true
I20260924 07:16:43.787746 22771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f5f661bd8dc40e0972223ed165e9d4e in term 2.
I20260924 07:16:43.788085 22565 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f5f661bd8dc40e0972223ed165e9d4e, 87db3c4cfe024dadb0995f9bf4f44719; no voters:
I20260924 07:16:43.788306 23301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 07:16:43.788362 23301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:43.788390 23301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:16:43.788681 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.788426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58298)
0924 07:16:43.788548 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.788614 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:16:43.788805 22565 leader_election.cc:334] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:43.789671 23301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:43.790011 23301 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 3 election: Requested vote from peers 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345), 87db3c4cfe024dadb0995f9bf4f44719 (127.15.127.188:45171)
I20260924 07:16:43.790377 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.790176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58304)
0924 07:16:43.790280 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.790324 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:16:43.790577 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "87db3c4cfe024dadb0995f9bf4f44719"
I20260924 07:16:43.790764 22771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:16:43.791117 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.790911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58308)
0924 07:16:43.791016 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.791068 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 07:16:43.791216 22565 leader_election.cc:334] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
I20260924 07:16:43.791937 22771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f5f661bd8dc40e0972223ed165e9d4e in term 3.
I20260924 07:16:43.792289 22565 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f5f661bd8dc40e0972223ed165e9d4e, 87db3c4cfe024dadb0995f9bf4f44719; no voters: 21b39edfe646474880d6855985f97bc5
I20260924 07:16:43.792464 23301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:16:43.792644 23301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [term 3 LEADER]: Becoming Leader. State: Replica: 2f5f661bd8dc40e0972223ed165e9d4e, State: Running, Role: LEADER
I20260924 07:16:43.792950 23301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } }
I20260924 07:16:43.794211 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5f661bd8dc40e0972223ed165e9d4e. Latest consensus state: current_term: 3 leader_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:43.794350 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:43.794651 23306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:43.796932 23306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:43.797407 23306 catalog_manager.cc:1259] Loaded cluster ID: 06e2379f12f84b1dab79dbd49b03d188
I20260924 07:16:43.797459 23306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:43.798035 23306 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:43.798434 23306 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Loaded TSK: 0
I20260924 07:16:43.799150 23306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:16:43.812991 22578 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:43.816406 22578 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:43.819830 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.819546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58318)
0924 07:16:43.819692 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.819754 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:16:43.820153 22771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [term 3 FOLLOWER]: Refusing update from remote peer 2f5f661bd8dc40e0972223ed165e9d4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 07:16:43.820611 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.820387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.0.0.1:58324)
0924 07:16:43.820490 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.820541 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 07:16:43.820690 23301 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260924 07:16:43.820895 22565 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e -> Peer 21b39edfe646474880d6855985f97bc5 (127.15.127.190:43345): Couldn't send request to peer 21b39edfe646474880d6855985f97bc5. Status: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:16:43.822328 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5f661bd8dc40e0972223ed165e9d4e. Latest consensus state: current_term: 3 leader_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:43.822494 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:43.823719 23304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:43.823858 23304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:43.825456 23308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "21b39edfe646474880d6855985f97bc5" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43345 } } peers { permanent_uuid: "2f5f661bd8dc40e0972223ed165e9d4e" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 44671 } attrs { promote: false } } peers { permanent_uuid: "87db3c4cfe024dadb0995f9bf4f44719" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 45171 } attrs { promote: false } } }
I20260924 07:16:43.825647 23308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87db3c4cfe024dadb0995f9bf4f44719 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:16:43.825759 23309 mvcc.cc:205] Tried to move back new op lower bound from 7332799298840539136 to 7332799298739032064. Current Snapshot: MvccSnapshot[applied={T|T < 7332799298840539136 or (T in {7332799298840539136})}]
I20260924 07:16:43.836032 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.831258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53246 (local address 127.15.127.130:38543)
0924 07:16:43.831515 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:43.831520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:43.831716 (+ 196us) server_negotiation.cc:407] Connection header received
0924 07:16:43.831918 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:43.831924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:43.831985 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:43.832044 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:43.832528 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.833138 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.834745 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.835326 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.835403 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.835445 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:43.835769 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 07:16:43.835874 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260924 07:16:43.836057 23302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.830766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:34001 (local address 127.0.0.1:41752)
0924 07:16:43.830915 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.830932 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:43.831086 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:43.831946 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:43.831949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:43.831970 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:43.832133 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.832141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.833504 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.833507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.834668 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.834677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.835638 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.835642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.835754 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.835777 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:16:43.835865 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:16:43.835937 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"spinlock_wait_cycles":2816}
I20260924 07:16:43.836118 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.831282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.15.127.131:34001)
0924 07:16:43.831664 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:43.831667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:43.831681 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:43.831728 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:43.831731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:43.831794 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:43.832357 (+ 563us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:43.832506 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.833380 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.834906 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.835527 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.835640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.835730 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:43.835882 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:16:43.835959 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":87,"threads_started":1}
I20260924 07:16:43.836653 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.831311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37253 (local address 127.0.0.1:40060)
0924 07:16:43.831915 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.831928 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:43.831997 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:43.832247 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:43.832250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:43.832262 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:43.832389 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.832395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.833827 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.833831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.835191 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.835201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.836194 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.836265 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.836357 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.836378 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:16:43.836450 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:16:43.836523 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":27,"thread_start_us":95,"threads_started":1}
I20260924 07:16:43.836720 23147 tablet_service.cc:1467] Processing CreateTablet for tablet 153c3713e3f9465f82d27de6588b2258 (DEFAULT_TABLE table=test-table [id=22a53ab7ed4c4444b47b4c3f70ee5052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:43.837101 23147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153c3713e3f9465f82d27de6588b2258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:43.837136 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.831651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.15.127.129:37253)
0924 07:16:43.831950 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:43.831955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:43.831971 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:43.832090 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:43.832093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:43.832142 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:43.832237 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:43.832510 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.833401 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.835343 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.835959 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.836082 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.836130 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:43.836846 (+ 716us) server_negotiation.cc:299] Negotiation successful
0924 07:16:43.836937 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
I20260924 07:16:43.836899 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.831124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38543 (local address 127.0.0.1:53246)
0924 07:16:43.831634 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.831647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:43.831789 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:43.832189 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:43.832191 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:43.832230 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:43.832389 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.832395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.833250 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.833258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.834544 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.834555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.835521 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.835524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.835625 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.835649 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:16:43.836691 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0924 07:16:43.836777 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":21,"spinlock_wait_cycles":1280,"thread_start_us":127,"threads_started":1}
I20260924 07:16:43.837875 22874 tablet_service.cc:1467] Processing CreateTablet for tablet 153c3713e3f9465f82d27de6588b2258 (DEFAULT_TABLE table=test-table [id=22a53ab7ed4c4444b47b4c3f70ee5052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:43.838135 22874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153c3713e3f9465f82d27de6588b2258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:43.838189 23011 tablet_service.cc:1467] Processing CreateTablet for tablet 153c3713e3f9465f82d27de6588b2258 (DEFAULT_TABLE table=test-table [id=22a53ab7ed4c4444b47b4c3f70ee5052]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:43.838413 23011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 153c3713e3f9465f82d27de6588b2258. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:43.840008 23318 tablet_bootstrap.cc:492] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Bootstrap starting.
I20260924 07:16:43.840581 23319 tablet_bootstrap.cc:492] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Bootstrap starting.
I20260924 07:16:43.841218 23320 tablet_bootstrap.cc:492] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56: Bootstrap starting.
I20260924 07:16:43.841265 23318 tablet_bootstrap.cc:654] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:43.841476 23319 tablet_bootstrap.cc:654] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:43.842228 23320 tablet_bootstrap.cc:654] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:43.842651 23318 tablet_bootstrap.cc:492] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: No bootstrap required, opened a new log
I20260924 07:16:43.842762 23318 ts_tablet_manager.cc:1397] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:43.842859 23319 tablet_bootstrap.cc:492] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: No bootstrap required, opened a new log
I20260924 07:16:43.842958 23319 ts_tablet_manager.cc:1397] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:43.843219 23318 raft_consensus.cc:348] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.843348 23318 raft_consensus.cc:374] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:43.843389 23318 raft_consensus.cc:729] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930a85194c5f41dbb7fdad053539fbce, State: Initialized, Role: FOLLOWER
I20260924 07:16:43.843420 23320 tablet_bootstrap.cc:492] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56: No bootstrap required, opened a new log
I20260924 07:16:43.843501 23320 ts_tablet_manager.cc:1397] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:43.843430 23319 raft_consensus.cc:348] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.843565 23319 raft_consensus.cc:374] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:43.843537 23318 consensus_queue.cc:260] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.843606 23319 raft_consensus.cc:729] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c315fa90cb245a9b4cfe62af1e739bb, State: Initialized, Role: FOLLOWER
I20260924 07:16:43.843768 23319 consensus_queue.cc:260] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.843938 23318 ts_tablet_manager.cc:1428] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:43.843974 23320 raft_consensus.cc:348] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.844079 23320 raft_consensus.cc:374] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:43.844123 23319 ts_tablet_manager.cc:1428] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 07:16:43.844133 23320 raft_consensus.cc:729] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c773f86301f403f83ff3271ace84f56, State: Initialized, Role: FOLLOWER
I20260924 07:16:43.844264 23320 consensus_queue.cc:260] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.844626 23320 ts_tablet_manager.cc:1428] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:43.845696 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.844917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:45333)
0924 07:16:43.845513 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.845580 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519}
I20260924 07:16:43.845983 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.845700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:42339)
0924 07:16:43.845854 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.845918 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:16:43.846565 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.844681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:40481)
0924 07:16:43.846423 (+ 1742us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.846496 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222}
I20260924 07:16:43.847504 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.847252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:36365)
0924 07:16:43.847387 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.847437 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 07:16:43.848383 23215 heartbeater.cc:503] Master 127.15.127.189:44671 was elected leader, sending a full tablet report...
I20260924 07:16:43.848496 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.848027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:43851)
0924 07:16:43.848312 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.848361 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
I20260924 07:16:43.848780 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.848556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:51551)
0924 07:16:43.848661 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.848711 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:16:43.848977 22942 heartbeater.cc:503] Master 127.15.127.189:44671 was elected leader, sending a full tablet report...
I20260924 07:16:43.850293 23079 heartbeater.cc:503] Master 127.15.127.189:44671 was elected leader, sending a full tablet report...
W20260924 07:16:43.866091 22943 tablet.cc:2369] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:43.923774 23324 raft_consensus.cc:482] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:43.923962 23324 raft_consensus.cc:504] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.925329 23324 leader_election.cc:288] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543), 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253)
I20260924 07:16:43.929961 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.925728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:56941 (local address 127.15.127.129:37253)
0924 07:16:43.925861 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:43.925866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:43.926107 (+ 241us) server_negotiation.cc:407] Connection header received
0924 07:16:43.926209 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:43.926212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:43.926265 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:43.926332 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:43.926584 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.927367 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.928885 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.929384 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.929530 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.929590 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:43.929767 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:16:43.929841 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:16:43.929977 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.925629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37253 (local address 127.15.127.131:56941)
0924 07:16:43.926010 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.926021 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:43.926098 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:43.926367 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:43.926370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:43.926388 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:43.926474 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.926486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.927499 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.927503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.928755 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.928765 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.929530 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.929534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.929636 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.929663 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 07:16:43.929751 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:16:43.929823 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":98,"thread_start_us":120,"threads_started":1}
I20260924 07:16:43.930127 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.925645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:56313 (local address 127.15.127.130:38543)
0924 07:16:43.925765 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:43.925772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:43.925793 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:16:43.925997 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:43.926000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:43.926045 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:43.926156 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:43.926472 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.927338 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.929123 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.929511 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.929683 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.929732 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:43.929926 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 07:16:43.930000 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:16:43.930130 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.925532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:38543 (local address 127.15.127.131:56313)
0924 07:16:43.925722 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.925739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:43.925848 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:43.926180 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:43.926184 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:43.926205 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:43.926352 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.926358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.927499 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.927506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.928759 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:43.928769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.929683 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.929686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:43.929781 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.929802 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:16:43.929908 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:16:43.929978 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":137}
I20260924 07:16:43.930488 22894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153c3713e3f9465f82d27de6588b2258" candidate_uuid: "930a85194c5f41dbb7fdad053539fbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c773f86301f403f83ff3271ace84f56" is_pre_election: true
I20260924 07:16:43.930619 23031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153c3713e3f9465f82d27de6588b2258" candidate_uuid: "930a85194c5f41dbb7fdad053539fbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" is_pre_election: true
I20260924 07:16:43.930749 23031 raft_consensus.cc:2454] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930a85194c5f41dbb7fdad053539fbce in term 0.
I20260924 07:16:43.930768 22894 raft_consensus.cc:2454] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930a85194c5f41dbb7fdad053539fbce in term 0.
I20260924 07:16:43.931131 23103 leader_election.cc:302] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c773f86301f403f83ff3271ace84f56, 930a85194c5f41dbb7fdad053539fbce; no voters:
I20260924 07:16:43.931372 23324 raft_consensus.cc:2781] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:43.931435 23324 raft_consensus.cc:482] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:43.931473 23324 raft_consensus.cc:3037] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:43.932500 23324 raft_consensus.cc:504] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.932893 23324 leader_election.cc:288] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [CANDIDATE]: Term 1 election: Requested vote from peers 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543), 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253)
I20260924 07:16:43.933282 22894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153c3713e3f9465f82d27de6588b2258" candidate_uuid: "930a85194c5f41dbb7fdad053539fbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c773f86301f403f83ff3271ace84f56"
I20260924 07:16:43.933292 23031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "153c3713e3f9465f82d27de6588b2258" candidate_uuid: "930a85194c5f41dbb7fdad053539fbce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c315fa90cb245a9b4cfe62af1e739bb"
I20260924 07:16:43.933411 23031 raft_consensus.cc:3037] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:43.933413 22894 raft_consensus.cc:3037] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:43.934595 23031 raft_consensus.cc:2454] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930a85194c5f41dbb7fdad053539fbce in term 1.
I20260924 07:16:43.934602 22894 raft_consensus.cc:2454] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930a85194c5f41dbb7fdad053539fbce in term 1.
I20260924 07:16:43.934928 23102 leader_election.cc:302] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c315fa90cb245a9b4cfe62af1e739bb, 930a85194c5f41dbb7fdad053539fbce; no voters:
I20260924 07:16:43.935123 23324 raft_consensus.cc:2781] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:43.935357 23324 raft_consensus.cc:686] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 1 LEADER]: Becoming Leader. State: Replica: 930a85194c5f41dbb7fdad053539fbce, State: Running, Role: LEADER
I20260924 07:16:43.935652 23324 consensus_queue.cc:237] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } }
I20260924 07:16:43.936127 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.935854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:42537)
0924 07:16:43.935982 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.936044 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:16:43.938189 22575 catalog_manager.cc:5495] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce reported cstate change: term changed from 0 to 1, leader changed from <none> to 930a85194c5f41dbb7fdad053539fbce (127.15.127.131). New cstate: current_term: 1 leader_uuid: "930a85194c5f41dbb7fdad053539fbce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "930a85194c5f41dbb7fdad053539fbce" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 } health_report { overall_health: UNKNOWN } } }
I20260924 07:16:43.961203 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.957506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.15.127.131:34001)
0924 07:16:43.957638 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:43.957643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:43.957658 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:43.957861 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:43.957864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:43.957922 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:43.958041 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:43.958290 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.958800 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.959668 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:43.959892 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:43.960000 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:43.960290 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:43.960767 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:43.960903 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:43.960965 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 07:16:43.961051 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:16:43.964116 23031 raft_consensus.cc:1261] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Refusing update from remote peer 930a85194c5f41dbb7fdad053539fbce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:43.964380 22894 raft_consensus.cc:1261] T 153c3713e3f9465f82d27de6588b2258 P 6c773f86301f403f83ff3271ace84f56 [term 1 FOLLOWER]: Refusing update from remote peer 930a85194c5f41dbb7fdad053539fbce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:43.964883 23324 consensus_queue.cc:1035] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c315fa90cb245a9b4cfe62af1e739bb" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:43.965219 23328 consensus_queue.cc:1035] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c773f86301f403f83ff3271ace84f56" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:43.966876 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.966562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:56137)
0924 07:16:43.966712 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.966784 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 07:16:43.967221 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.966985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:58173)
0924 07:16:43.967095 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.967149 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:16:43.972242 23298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.971916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.129:42447)
0924 07:16:43.972091 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.972156 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 07:16:43.973202 23299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.970478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:43483)
0924 07:16:43.973060 (+ 2582us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.973123 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2486}
I20260924 07:16:43.973887 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.973670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.131:32783)
0924 07:16:43.973765 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.973823 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:16:43.975507 23334 mvcc.cc:205] Tried to move back new op lower bound from 7332799299431694336 to 7332799299323432960. Current Snapshot: MvccSnapshot[applied={T|T < 7332799299431694336}]
I20260924 07:16:43.978058 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:43.977780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43345 (local address 127.15.127.130:59915)
0924 07:16:43.977924 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.977982 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:43345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 07:16:43.979930 22577 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:43.980132 22577 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:43.984577 22577 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f5f661bd8dc40e0972223ed165e9d4e: Sending DeleteTablet for 3 replicas of tablet 153c3713e3f9465f82d27de6588b2258
I20260924 07:16:43.985442 23011 tablet_service.cc:1514] Processing DeleteTablet for tablet 153c3713e3f9465f82d27de6588b2258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:43 UTC) from {username='oryx', principal='oryx/127.15.127.189@KRBTEST.COM'} at 127.0.0.1:53246
I20260924 07:16:43.985868 23342 tablet_replica.cc:330] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: stopping tablet replica
I20260924 07:16:43.986112 23342 raft_consensus.cc:2229] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:43.986187 23147 tablet_service.cc:1514] Processing DeleteTablet for tablet 153c3713e3f9465f82d27de6588b2258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:43 UTC) from {username='oryx', principal='oryx/127.15.127.189@KRBTEST.COM'} at 127.0.0.1:41752
I20260924 07:16:43.986611 22874 tablet_service.cc:1514] Processing DeleteTablet for tablet 153c3713e3f9465f82d27de6588b2258 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:43 UTC) from {username='oryx', principal='oryx/127.15.127.189@KRBTEST.COM'} at 127.0.0.1:40060
I20260924 07:16:43.986899 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22722
I20260924 07:16:43.988929 23342 raft_consensus.cc:2258] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:43.989848 23342 ts_tablet_manager.cc:1905] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:43.993404 23342 ts_tablet_manager.cc:1918] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:43.993510 23342 log.cc:1196] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/153c3713e3f9465f82d27de6588b2258
I20260924 07:16:43.993819 23342 ts_tablet_manager.cc:1939] T 153c3713e3f9465f82d27de6588b2258 P 5c315fa90cb245a9b4cfe62af1e739bb: Deleting consensus metadata
I20260924 07:16:43.993831 23343 tablet_replica.cc:330] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: stopping tablet replica
I20260924 07:16:43.994119 23343 raft_consensus.cc:2229] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:43.994520 23343 raft_consensus.cc:2258] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:43.994747 22564 catalog_manager.cc:4841] TS 5c315fa90cb245a9b4cfe62af1e739bb (127.15.127.130:38543): tablet 153c3713e3f9465f82d27de6588b2258 (table test-table [id=22a53ab7ed4c4444b47b4c3f70ee5052]) successfully deleted
I20260924 07:16:43.995369 23343 ts_tablet_manager.cc:1905] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:16:43.996711 23313 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:43.996413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37253 (local address 127.0.0.1:40072)
0924 07:16:43.996528 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:43.996595 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:37253: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:16:43.996907 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22948
W20260924 07:16:43.996901 22565 catalog_manager.cc:4584] TS 6c773f86301f403f83ff3271ace84f56 (127.15.127.129:37253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 153c3713e3f9465f82d27de6588b2258: Network error: Client connection negotiation failed: client connection to 127.15.127.129:37253: connect: Connection reset by peer (error 104)
I20260924 07:16:43.999426 23343 ts_tablet_manager.cc:1918] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:43.999518 23343 log.cc:1196] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/153c3713e3f9465f82d27de6588b2258
I20260924 07:16:43.999794 23343 ts_tablet_manager.cc:1939] T 153c3713e3f9465f82d27de6588b2258 P 930a85194c5f41dbb7fdad053539fbce: Deleting consensus metadata
I20260924 07:16:44.000782 22565 catalog_manager.cc:4841] TS 930a85194c5f41dbb7fdad053539fbce (127.15.127.131:34001): tablet 153c3713e3f9465f82d27de6588b2258 (table test-table [id=22a53ab7ed4c4444b47b4c3f70ee5052]) successfully deleted
I20260924 07:16:44.003372 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 23084
I20260924 07:16:44.010102 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22440
I20260924 07:16:44.017190 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 22544
2026-09-24T07:16:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7128 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:44 dist-test-slave-zx46 krb5kdc[23347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:44 dist-test-slave-zx46 krb5kdc[23347](info): set up 2 sockets
Sep 24 07:16:44 dist-test-slave-zx46 krb5kdc[23347](info): commencing operation
krb5kdc: starting...
W20260924 07:16:45.081765 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab.
Entry for principal oryx/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab.
I20260924 07:16:45.209924 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36169
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab
--principal=oryx/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:36169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:45.326876 23360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:45.327171 23360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:45.327214 23360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:45.331321 23360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:45.331386 23360 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:16:45.331403 23360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:45.331422 23360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:45.331439 23360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:45.331454 23360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:45.336814 23360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:36169
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.190.keytab
--principal=oryx/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:45.337962 23360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:45.338965 23360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:45.347973 23366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:45.348210 23365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:45.347990 23368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:45.348111 23360 server_base.cc:1034] running on GCE node
I20260924 07:16:45.348899 23360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:45.350178 23360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:45.351359 23360 hybrid_clock.cc:648] HybridClock initialized: now 1790234205351323 us; error 54 us; skew 500 ppm
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:45.354358 23360 init.cc:377] Logged in from keytab as oryx/127.15.127.190@KRBTEST.COM (short username oryx)
I20260924 07:16:45.355360 23360 webserver.cc:460] Webserver started at http://127.15.127.190:40937/ using document root <none> and password file <none>
I20260924 07:16:45.355950 23360 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:45.356043 23360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:45.356284 23360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:45.358486 23360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "25f368d115a0422ca941549e2a942b93"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "03b131d102c846408470ce42ffdfb139"
server_key_iv: "995ea57565891c3677f644c3eb70b9f9"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.359099 23360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "25f368d115a0422ca941549e2a942b93"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "03b131d102c846408470ce42ffdfb139"
server_key_iv: "995ea57565891c3677f644c3eb70b9f9"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.363046 23360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 07:16:45.365612 23375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:45.367074 23360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:45.367246 23360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "25f368d115a0422ca941549e2a942b93"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "03b131d102c846408470ce42ffdfb139"
server_key_iv: "995ea57565891c3677f644c3eb70b9f9"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.367365 23360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:45.383850 23360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:45.387305 23360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:45.387533 23360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:45.396427 23360 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:36169
I20260924 07:16:45.396437 23437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:36169 every 8 connection(s)
I20260924 07:16:45.397487 23360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:45.400481 23438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:45.406512 23438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Bootstrap starting.
I20260924 07:16:45.407073 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 23360
I20260924 07:16:45.407215 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:45.407516 15870 external_mini_cluster.cc:1428] Setting key 299b1bfb28e26c6aae5ae468d5f59b13
I20260924 07:16:45.409708 23438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:45.410507 23438 log.cc:824] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:45.412554 23438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: No bootstrap required, opened a new log
I20260924 07:16:45.416512 23438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f368d115a0422ca941549e2a942b93" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36169 } }
I20260924 07:16:45.416867 23438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:45.416934 23438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f368d115a0422ca941549e2a942b93, State: Initialized, Role: FOLLOWER
I20260924 07:16:45.417452 23438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f368d115a0422ca941549e2a942b93" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36169 } }
I20260924 07:16:45.417582 23438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:45.417677 23438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:45.417845 23438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:45.419028 23438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f368d115a0422ca941549e2a942b93" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36169 } }
I20260924 07:16:45.419395 23438 leader_election.cc:302] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25f368d115a0422ca941549e2a942b93; no voters:
I20260924 07:16:45.419723 23438 leader_election.cc:288] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:45.419826 23443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:45.420094 23443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [term 1 LEADER]: Becoming Leader. State: Replica: 25f368d115a0422ca941549e2a942b93, State: Running, Role: LEADER
I20260924 07:16:45.420789 23443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f368d115a0422ca941549e2a942b93" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36169 } }
I20260924 07:16:45.421130 23438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:45.422636 23445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25f368d115a0422ca941549e2a942b93. Latest consensus state: current_term: 1 leader_uuid: "25f368d115a0422ca941549e2a942b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f368d115a0422ca941549e2a942b93" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36169 } } }
I20260924 07:16:45.422863 23445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:45.423235 23444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25f368d115a0422ca941549e2a942b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25f368d115a0422ca941549e2a942b93" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36169 } } }
I20260924 07:16:45.423280 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.409116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50768 (local address 127.15.127.190:36169)
0924 07:16:45.409622 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:45.409639 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:45.409680 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:16:45.410438 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:45.410457 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:45.410886 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:45.411287 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:45.412105 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.413241 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.414112 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.414605 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.414791 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.417571 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:45.417580 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:45.417602 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:45.417635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:45.419535 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.420267 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.420274 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.420281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.420422 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.420712 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.420716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.420718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.421315 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:45.421514 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:45.421824 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 07:16:45.422189 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":155,"threads_started":1}
I20260924 07:16:45.423769 23457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:45.424929 23444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93 [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:45.426558 23457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:45.432076 23457 catalog_manager.cc:1347] Generated new cluster ID: 4b8cd4fd953e47f2940d1da916a90b2c
I20260924 07:16:45.432163 23457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:45.458173 23457 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:45.459390 23457 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:45.465049 23457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Generated new TSK 0
I20260924 07:16:45.465788 23457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab.
Entry for principal oryx/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab.
I20260924 07:16:45.501456 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:36169
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:45.622393 23464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:45.622725 23464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:45.622824 23464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:45.626982 23464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:45.627048 23464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:45.627087 23464 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:45.627192 23464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:45.632582 23464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:36169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:45.633702 23464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:45.634716 23464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:45.644992 23470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:45.645022 23469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:45.644996 23472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:45.645174 23464 server_base.cc:1034] running on GCE node
I20260924 07:16:45.645781 23464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:45.646484 23464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:45.647687 23464 hybrid_clock.cc:648] HybridClock initialized: now 1790234205647672 us; error 29 us; skew 500 ppm
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:45.650605 23464 init.cc:377] Logged in from keytab as oryx/127.15.127.129@KRBTEST.COM (short username oryx)
I20260924 07:16:45.651549 23464 webserver.cc:460] Webserver started at http://127.15.127.129:36593/ using document root <none> and password file <none>
I20260924 07:16:45.652155 23464 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:45.652251 23464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:45.652501 23464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:45.654798 23464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "aca2034c82a14ce6aa8184b72e860316"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "ff39fb8c7d19fc96ea013749c151d76d"
server_key_iv: "defc05c53063af9c697e343d3e0ca732"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.655395 23464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "aca2034c82a14ce6aa8184b72e860316"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "ff39fb8c7d19fc96ea013749c151d76d"
server_key_iv: "defc05c53063af9c697e343d3e0ca732"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.659339 23464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 07:16:45.661988 23479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:45.663374 23464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:45.663566 23464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "aca2034c82a14ce6aa8184b72e860316"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "ff39fb8c7d19fc96ea013749c151d76d"
server_key_iv: "defc05c53063af9c697e343d3e0ca732"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.663705 23464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:45.689198 23464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:45.692564 23464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:45.692814 23464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:45.694475 23464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:45.694574 23464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:45.694669 23464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:45.694723 23464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:45.707408 23464 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:43449
I20260924 07:16:45.707475 23594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:43449 every 8 connection(s)
I20260924 07:16:45.708806 23464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:45.710970 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.695580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:48763 (local address 127.15.127.190:36169)
0924 07:16:45.695715 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:45.695719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:45.696991 (+ 1272us) server_negotiation.cc:407] Connection header received
0924 07:16:45.698310 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:45.698324 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:45.698405 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:45.698489 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:45.700344 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.701252 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.702823 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.703100 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.703251 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.706638 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:45.706646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:45.706653 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:45.706693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:45.708602 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.709260 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.709267 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.709272 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.709391 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.709854 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.709858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.709860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.710119 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:45.710210 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:45.710730 (+ 520us) server_negotiation.cc:299] Negotiation successful
0924 07:16:45.710851 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:16:45.711895 23488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.696241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.15.127.129:48763)
0924 07:16:45.696796 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:45.696845 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:45.698015 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:45.698729 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:45.698741 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:45.699344 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:45.700169 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.700193 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.701394 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.701398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.702682 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.702693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.703315 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.703321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.703594 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.704399 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:45.704422 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:45.706434 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:45.708734 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.708739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.708750 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.709148 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.709517 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.709524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.709529 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.709730 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.710228 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:45.710234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:45.710476 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 07:16:45.710641 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 07:16:45.710915 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":148,"threads_started":1}
I20260924 07:16:45.713481 23595 heartbeater.cc:347] Connected to a master server at 127.15.127.190:36169
I20260924 07:16:45.713661 23595 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:45.714171 23595 heartbeater.cc:511] Master 127.15.127.190:36169 requested a full tablet report, sending...
I20260924 07:16:45.715845 23391 ts_manager.cc:194] Registered new tserver with Master: aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449)
I20260924 07:16:45.717343 23391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:48763
I20260924 07:16:45.718917 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 23464
I20260924 07:16:45.719059 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:45.719410 15870 external_mini_cluster.cc:1428] Setting key d513d1a65733d6bcc02b1d63eb7bfd47
I20260924 07:16:45.724431 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.715723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.15.127.129:39273)
0924 07:16:45.716175 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:45.716191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:45.716293 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:45.716550 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:45.716553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:45.716773 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:45.716892 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.716899 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.717854 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.717857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.718655 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.718665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.719153 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.719157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.719247 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.719829 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:45.719841 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:45.720438 (+ 597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:45.722727 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.722729 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.722731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.723043 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.723382 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.723385 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.723387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.723479 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.723861 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:45.723864 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:45.723967 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 07:16:45.724077 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:16:45.724237 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":120,"threads_started":1}
I20260924 07:16:45.724423 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.715868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:39273 (local address 127.15.127.190:36169)
0924 07:16:45.716004 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:45.716009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:45.716273 (+ 264us) server_negotiation.cc:407] Connection header received
0924 07:16:45.716380 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:45.716382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:45.716426 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:45.716508 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:45.717063 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.717719 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.718775 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.719050 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.719148 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.720665 (+ 1517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:45.720668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:45.720671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:45.720703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:45.722598 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.723170 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.723192 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.723196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.723281 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.723577 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.723580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.723582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.723763 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:45.723872 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:45.724130 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 07:16:45.724257 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
WARNING: no policy specified for oryx/127.15.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab.
Entry for principal oryx/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab.
I20260924 07:16:45.748657 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:36169
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:45.865741 23604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:45.866039 23604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:45.866122 23604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:45.870574 23604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:45.870653 23604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:45.870702 23604 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:45.870782 23604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:45.876570 23604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:36169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:45.877805 23604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:45.878916 23604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:45.889946 23609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:45.890043 23604 server_base.cc:1034] running on GCE node
W20260924 07:16:45.889950 23612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:45.890286 23610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:45.890722 23604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:45.891443 23604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:45.892714 23604 hybrid_clock.cc:648] HybridClock initialized: now 1790234205892693 us; error 33 us; skew 500 ppm
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:45.895784 23604 init.cc:377] Logged in from keytab as oryx/127.15.127.130@KRBTEST.COM (short username oryx)
I20260924 07:16:45.896812 23604 webserver.cc:460] Webserver started at http://127.15.127.130:43103/ using document root <none> and password file <none>
I20260924 07:16:45.897473 23604 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:45.897575 23604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:45.897819 23604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:45.899959 23604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "035c2d6231b141109fee34e97cde22d4"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "c1441fee53395fd1891994ccf975f41b"
server_key_iv: "7e139ba15171aeb4e73f30ed370d370d"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.900559 23604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "035c2d6231b141109fee34e97cde22d4"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "c1441fee53395fd1891994ccf975f41b"
server_key_iv: "7e139ba15171aeb4e73f30ed370d370d"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.904567 23604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:16:45.907385 23619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:45.908757 23604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:45.908913 23604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "035c2d6231b141109fee34e97cde22d4"
format_stamp: "Formatted at 2026-09-24 07:16:45 on dist-test-slave-zx46"
server_key: "c1441fee53395fd1891994ccf975f41b"
server_key_iv: "7e139ba15171aeb4e73f30ed370d370d"
server_key_version: "encryptionkey@0"
I20260924 07:16:45.909116 23604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:45.928426 23604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:45.931964 23604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:45.932245 23604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:45.933916 23604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:45.933990 23604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:45.934073 23604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:45.934109 23604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:16:45 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:45.945245 23604 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:40951
I20260924 07:16:45.945302 23734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:40951 every 8 connection(s)
I20260924 07:16:45.946465 23604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:45.948596 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.934766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:38617 (local address 127.15.127.190:36169)
0924 07:16:45.934946 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:45.934951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:45.935837 (+ 886us) server_negotiation.cc:407] Connection header received
0924 07:16:45.936884 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:45.936890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:45.936958 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:45.937061 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:45.938575 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.939187 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.940395 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.940623 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.940746 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.944494 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:45.944501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:45.944504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:45.944549 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:45.946401 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.947065 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.947068 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.947070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.947165 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.947535 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.947538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.947539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.947730 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:45.947812 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:45.948352 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 07:16:45.948460 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:16:45.949630 23628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.935188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.15.127.130:38617)
0924 07:16:45.935666 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:45.935719 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:45.936621 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:45.937511 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:45.937518 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:45.937900 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:45.938424 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.938439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.939325 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.939329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.940259 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.940269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.940797 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.940801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.940977 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.941778 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:45.941803 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:45.944289 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:45.946549 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.946557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.946578 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.946938 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.947270 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.947273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.947276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.947423 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.947843 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:45.947848 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:45.948089 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 07:16:45.948288 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 07:16:45.948557 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":135,"threads_started":1}
I20260924 07:16:45.950973 23735 heartbeater.cc:347] Connected to a master server at 127.15.127.190:36169
I20260924 07:16:45.951251 23735 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:45.951857 23735 heartbeater.cc:511] Master 127.15.127.190:36169 requested a full tablet report, sending...
I20260924 07:16:45.953199 23392 ts_manager.cc:194] Registered new tserver with Master: 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951)
I20260924 07:16:45.954069 23392 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:38617
I20260924 07:16:45.956081 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 23604
I20260924 07:16:45.956198 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:45.956519 15870 external_mini_cluster.cc:1428] Setting key eb6e35c4791375fba333bee6d35fde31
I20260924 07:16:45.961410 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.953647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.15.127.130:60193)
0924 07:16:45.953930 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:45.953943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:45.954027 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:45.954338 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:45.954341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:45.954605 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:45.954739 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.954746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.955517 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.955529 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.956327 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:45.956338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.956831 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.956835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:45.956929 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.957526 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:45.957538 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:45.957954 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:45.959824 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.959826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.959828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.960015 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.960315 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:45.960320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:45.960322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:45.960408 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:45.960908 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:45.960911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:45.960998 (+ 87us) client_negotiation.cc:769] Sending connection context
0924 07:16:45.961152 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:16:45.961271 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":103,"threads_started":1}
I20260924 07:16:45.961628 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:45.953756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:60193 (local address 127.15.127.190:36169)
0924 07:16:45.953923 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:45.953927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:45.954015 (+ 88us) server_negotiation.cc:407] Connection header received
0924 07:16:45.954094 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:45.954097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:45.954144 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:45.954224 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:45.954856 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.955373 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.956441 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:45.956714 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:45.956797 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:45.958084 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:45.958088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:45.958091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:45.958114 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:45.959584 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.960122 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.960126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.960128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.960219 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:45.960507 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:45.960510 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:45.960512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:45.960806 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:45.960910 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:45.961239 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 07:16:45.961380 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
WARNING: no policy specified for oryx/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab.
Entry for principal oryx/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab.
I20260924 07:16:45.984581 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:36169
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:46.103561 23744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:46.103863 23744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:46.103909 23744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:46.108150 23744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:46.108234 23744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:46.108280 23744 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:16:46.108314 23744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:46.114089 23744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.127.190:36169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:46.115324 23744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:46.116434 23744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:46.126592 23752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:46.126598 23750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:46.126770 23744 server_base.cc:1034] running on GCE node
W20260924 07:16:46.126940 23749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:46.127471 23744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:46.128189 23744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:46.129415 23744 hybrid_clock.cc:648] HybridClock initialized: now 1790234206129396 us; error 30 us; skew 500 ppm
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234206, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:46.132956 23744 init.cc:377] Logged in from keytab as oryx/127.15.127.131@KRBTEST.COM (short username oryx)
I20260924 07:16:46.134095 23744 webserver.cc:460] Webserver started at http://127.15.127.131:37433/ using document root <none> and password file <none>
I20260924 07:16:46.134752 23744 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:46.134852 23744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:46.135099 23744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:46.137396 23744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
format_stamp: "Formatted at 2026-09-24 07:16:46 on dist-test-slave-zx46"
server_key: "158ee54f897c6b49e0991b4255c98b47"
server_key_iv: "e662e76dfffca2c2c15298d5da8a774d"
server_key_version: "encryptionkey@0"
I20260924 07:16:46.138002 23744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
format_stamp: "Formatted at 2026-09-24 07:16:46 on dist-test-slave-zx46"
server_key: "158ee54f897c6b49e0991b4255c98b47"
server_key_iv: "e662e76dfffca2c2c15298d5da8a774d"
server_key_version: "encryptionkey@0"
I20260924 07:16:46.142187 23744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:16:46.144922 23759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:46.146272 23744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:46.146440 23744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
format_stamp: "Formatted at 2026-09-24 07:16:46 on dist-test-slave-zx46"
server_key: "158ee54f897c6b49e0991b4255c98b47"
server_key_iv: "e662e76dfffca2c2c15298d5da8a774d"
server_key_version: "encryptionkey@0"
I20260924 07:16:46.146600 23744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:46.164649 23744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:46.168298 23744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:46.168521 23744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:46.170090 23744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:46.170171 23744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:46.170244 23744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:46.170288 23744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:46.180318 23744 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:45321
I20260924 07:16:46.181198 23874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:45321 every 8 connection(s)
I20260924 07:16:46.181504 23744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:46.182003 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 23744
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234206, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:46.182139 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:46.182469 15870 external_mini_cluster.cc:1428] Setting key 3fa4cf65a3564163cab331687fe3a16d
I20260924 07:16:46.187687 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.171218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:35351 (local address 127.15.127.190:36169)
0924 07:16:46.171359 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.171398 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.172572 (+ 1174us) server_negotiation.cc:407] Connection header received
0924 07:16:46.174034 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.174039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.174100 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.174197 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.176254 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.177227 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.178942 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.179366 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.179507 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.183103 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:46.183106 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:46.183108 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:46.183137 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:46.185072 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.185669 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.185673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.185675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.185774 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.186388 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.186394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.186396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.186651 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:46.186768 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.187378 (+ 610us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.187516 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:16:46.188604 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.171844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.15.127.131:35351)
0924 07:16:46.172404 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.172451 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.173722 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.174486 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.174501 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.175132 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:46.176029 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.176053 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.177355 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.177359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.178813 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.178826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.179553 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.179558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.179777 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.180524 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:46.180546 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:46.182822 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:46.185213 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:46.185220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:46.185237 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:46.185540 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:46.185884 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:46.185888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:46.185890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:46.186066 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:46.186777 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:46.186787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:46.187077 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.187341 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.187700 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":158,"threads_started":1}
I20260924 07:16:46.189913 23875 heartbeater.cc:347] Connected to a master server at 127.15.127.190:36169
I20260924 07:16:46.190081 23875 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:46.190670 23875 heartbeater.cc:511] Master 127.15.127.190:36169 requested a full tablet report, sending...
I20260924 07:16:46.191831 23391 ts_manager.cc:194] Registered new tserver with Master: 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.192672 23391 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:35351
I20260924 07:16:46.196564 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:16:46.200397 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.192229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.15.127.131:56533)
0924 07:16:46.192569 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.192583 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.192696 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.192976 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.192981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.193226 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:46.193364 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.193372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.194086 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.194090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.194894 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.194903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.195392 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.195397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.195478 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.196123 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:46.196135 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:46.196462 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:46.198610 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:46.198613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:46.198614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:46.198955 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:46.199246 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:46.199251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:46.199255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:46.199361 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:46.199876 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:46.199879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:46.199995 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.200116 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.200254 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260924 07:16:46.200497 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.192306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:56533 (local address 127.15.127.190:36169)
0924 07:16:46.192430 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.192434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.192662 (+ 228us) server_negotiation.cc:407] Connection header received
0924 07:16:46.192803 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.192809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.192857 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.192924 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.193465 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.193960 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.195005 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.195284 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.195366 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.196603 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:46.196607 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:46.196610 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:46.196633 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:46.198474 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.199064 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.199067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.199068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.199145 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.199574 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.199577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.199579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.199772 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:46.199864 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.200215 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.200330 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234206, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:16:46.224680 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:46.217524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46322 (local address 127.15.127.190:36169)
0924 07:16:46.217689 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.217693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.217706 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:46.217750 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.217753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.217793 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.217868 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.218654 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.219175 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.220186 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.220403 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.220485 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.224520 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46322: BlockingRecv error: recv got EOF from 127.0.0.1:46322 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234206, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:46.234828 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.226250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.15.127.190:36169)
0924 07:16:46.226373 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.226376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.226571 (+ 195us) server_negotiation.cc:407] Connection header received
0924 07:16:46.226732 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.226735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.226786 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.226867 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.227439 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.228004 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.228795 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.229154 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.229247 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.231774 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:46.231778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:46.231780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:46.231809 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:46.233324 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.233784 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.233787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.233788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.233868 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.234097 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.234099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.234100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.234293 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:46.234396 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.234600 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.234706 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:16:46.238113 23391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:46.241540 23391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:46.257884 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.251485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:40951 (local address 127.0.0.1:42488)
0924 07:16:46.252403 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.252466 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.252703 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.253513 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.253518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.253537 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.253696 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.253703 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.254750 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.254752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.256186 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.256196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.257304 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.257310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.257422 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.257460 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.257583 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.257758 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":31,"thread_start_us":156,"threads_started":1}
I20260924 07:16:46.257891 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.251661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42488 (local address 127.15.127.130:40951)
0924 07:16:46.252128 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.252142 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.252559 (+ 417us) server_negotiation.cc:407] Connection header received
0924 07:16:46.252917 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.252926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.253242 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.253350 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.253827 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.254624 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.256322 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.257175 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.257358 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.257501 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.257609 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.257724 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":105,"threads_started":1}
I20260924 07:16:46.258163 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.251906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43449 (local address 127.0.0.1:46054)
0924 07:16:46.252641 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.252654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.252764 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.253782 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.253785 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.253803 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.253902 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.253907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.255734 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.255737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.256691 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.256702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.257771 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.257776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.257883 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.257901 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.257945 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.258007 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":3,"thread_start_us":96,"threads_started":1}
I20260924 07:16:46.258585 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.252383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:45321 (local address 127.0.0.1:38750)
0924 07:16:46.253153 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.253167 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.253264 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.253702 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.253704 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.253721 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.253822 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.253828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.255181 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.255184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.256376 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.256386 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.257567 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.257571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.257716 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.257753 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.257816 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.258466 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":562,"thread_start_us":107,"threads_started":1}
I20260924 07:16:46.259032 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.252874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.15.127.131:45321)
0924 07:16:46.253198 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.253213 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.253237 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:16:46.253369 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.253374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.253558 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.253729 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.253909 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.255062 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.256502 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.257227 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.257363 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.257501 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.257904 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.258004 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":120,"threads_started":1}
I20260924 07:16:46.259128 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.252100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.15.127.129:43449)
0924 07:16:46.253397 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.253404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.253424 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:16:46.253489 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.253493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.253646 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.254048 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.254176 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.255597 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.256820 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.257638 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.258568 (+ 930us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.258845 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.258896 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.258997 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":85,"threads_started":1}
I20260924 07:16:46.261215 23669 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 (DEFAULT_TABLE table=test-table [id=1765d9dfc8f942b1833efbafe04c89a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.261746 23809 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 (DEFAULT_TABLE table=test-table [id=1765d9dfc8f942b1833efbafe04c89a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.261749 23529 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 (DEFAULT_TABLE table=test-table [id=1765d9dfc8f942b1833efbafe04c89a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.262430 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.262799 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.262794 23529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.269863 23903 tablet_bootstrap.cc:492] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Bootstrap starting.
I20260924 07:16:46.271220 23904 tablet_bootstrap.cc:492] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Bootstrap starting.
I20260924 07:16:46.272279 23903 tablet_bootstrap.cc:654] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.273164 23903 log.cc:824] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:46.273231 23904 tablet_bootstrap.cc:654] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.273442 23905 tablet_bootstrap.cc:492] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Bootstrap starting.
I20260924 07:16:46.274065 23904 log.cc:824] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:46.275584 23905 tablet_bootstrap.cc:654] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.275867 23903 tablet_bootstrap.cc:492] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: No bootstrap required, opened a new log
I20260924 07:16:46.276083 23903 ts_tablet_manager.cc:1397] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 07:16:46.276332 23904 tablet_bootstrap.cc:492] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: No bootstrap required, opened a new log
I20260924 07:16:46.276413 23905 log.cc:824] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:46.276525 23904 ts_tablet_manager.cc:1397] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:16:46.278628 23905 tablet_bootstrap.cc:492] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: No bootstrap required, opened a new log
I20260924 07:16:46.278828 23905 ts_tablet_manager.cc:1397] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:16:46.279572 23903 raft_consensus.cc:348] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.279846 23903 raft_consensus.cc:374] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.279901 23903 raft_consensus.cc:729] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.280364 23903 consensus_queue.cc:260] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.281283 23904 raft_consensus.cc:348] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.281512 23903 ts_tablet_manager.cc:1428] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 07:16:46.281575 23904 raft_consensus.cc:374] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.281665 23904 raft_consensus.cc:729] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.281893 23735 heartbeater.cc:503] Master 127.15.127.190:36169 was elected leader, sending a full tablet report...
I20260924 07:16:46.282179 23904 consensus_queue.cc:260] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.282819 23595 heartbeater.cc:503] Master 127.15.127.190:36169 was elected leader, sending a full tablet report...
I20260924 07:16:46.282883 23905 raft_consensus.cc:348] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.283145 23905 raft_consensus.cc:374] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.283211 23905 raft_consensus.cc:729] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801e3134ed4048908ed3dc3acc3ddcb9, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.283622 23905 consensus_queue.cc:260] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.284214 23904 ts_tablet_manager.cc:1428] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260924 07:16:46.284720 23905 ts_tablet_manager.cc:1428] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:16:46.284766 23875 heartbeater.cc:503] Master 127.15.127.190:36169 was elected leader, sending a full tablet report...
I20260924 07:16:46.328403 23910 raft_consensus.cc:482] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:46.328604 23910 raft_consensus.cc:504] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.329896 23910 leader_election.cc:288] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.334105 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.330696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:57195 (local address 127.15.127.131:45321)
0924 07:16:46.330841 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.330846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.330945 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:16:46.331049 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.331052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.331093 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.331171 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.331441 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.331933 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.333126 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.333545 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.333668 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.333730 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.333889 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.333972 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:16:46.334148 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.330238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60927 (local address 127.15.127.130:40951)
0924 07:16:46.330362 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.330366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.330613 (+ 247us) server_negotiation.cc:407] Connection header received
0924 07:16:46.330885 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.330893 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.330949 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.331026 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.331348 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.331903 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.333141 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.333558 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.333688 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.333752 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.333968 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.334042 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:16:46.334125 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.330549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:45321 (local address 127.15.127.129:57195)
0924 07:16:46.330852 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.330866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.330964 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.331189 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.331196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.331210 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.331333 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.331339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.332068 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.332071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.332968 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.332977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.333661 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.333665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.333760 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.333789 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.333862 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.333944 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":61,"thread_start_us":104,"threads_started":1}
I20260924 07:16:46.334149 23912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.330102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:40951 (local address 127.15.127.129:60927)
0924 07:16:46.330511 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.330523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.330697 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.331048 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.331051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.331071 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.331223 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.331231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.332015 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.332022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.332964 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.332973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.333696 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.333699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.333817 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.333855 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.333934 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.333998 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":33,"thread_start_us":139,"threads_started":1}
I20260924 07:16:46.334894 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b5e1de02b4dffb1a54a1ced3143c5" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" is_pre_election: true
I20260924 07:16:46.334895 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b5e1de02b4dffb1a54a1ced3143c5" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035c2d6231b141109fee34e97cde22d4" is_pre_election: true
I20260924 07:16:46.335210 23829 raft_consensus.cc:2454] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 0.
I20260924 07:16:46.335223 23689 raft_consensus.cc:2454] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 0.
I20260924 07:16:46.335741 23480 leader_election.cc:302] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, aca2034c82a14ce6aa8184b72e860316; no voters:
I20260924 07:16:46.336058 23910 raft_consensus.cc:2781] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:46.336144 23910 raft_consensus.cc:482] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:46.336179 23910 raft_consensus.cc:3037] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.337301 23910 raft_consensus.cc:504] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.337654 23910 leader_election.cc:288] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 election: Requested vote from peers 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.338064 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b5e1de02b4dffb1a54a1ced3143c5" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035c2d6231b141109fee34e97cde22d4"
I20260924 07:16:46.338095 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9b5e1de02b4dffb1a54a1ced3143c5" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
I20260924 07:16:46.338179 23689 raft_consensus.cc:3037] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.338232 23829 raft_consensus.cc:3037] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.339274 23689 raft_consensus.cc:2454] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 1.
I20260924 07:16:46.339352 23829 raft_consensus.cc:2454] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 1.
I20260924 07:16:46.339609 23480 leader_election.cc:302] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, aca2034c82a14ce6aa8184b72e860316; no voters:
I20260924 07:16:46.339804 23910 raft_consensus.cc:2781] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:46.340044 23910 raft_consensus.cc:686] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 1 LEADER]: Becoming Leader. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Running, Role: LEADER
I20260924 07:16:46.340384 23910 consensus_queue.cc:237] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } }
I20260924 07:16:46.343753 23390 catalog_manager.cc:5495] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 reported cstate change: term changed from 0 to 1, leader changed from <none> to aca2034c82a14ce6aa8184b72e860316 (127.15.127.129). New cstate: current_term: 1 leader_uuid: "aca2034c82a14ce6aa8184b72e860316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } health_report { overall_health: HEALTHY } } }
I20260924 07:16:46.358023 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.354274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.15.127.129:43449)
0924 07:16:46.354408 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.354412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.354428 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:46.354623 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.354625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.354703 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.354793 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:46.355051 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.355550 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.356466 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.356707 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.356827 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.357146 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.357621 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.357760 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.357802 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.357863 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:16:46.364677 23689 raft_consensus.cc:1261] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Refusing update from remote peer aca2034c82a14ce6aa8184b72e860316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.364912 23829 raft_consensus.cc:1261] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Refusing update from remote peer aca2034c82a14ce6aa8184b72e860316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.365582 23914 consensus_queue.cc:1035] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.365808 23910 consensus_queue.cc:1035] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.374185 23917 mvcc.cc:205] Tried to move back new op lower bound from 7332799309260656640 to 7332799309172936704. Current Snapshot: MvccSnapshot[applied={T|T < 7332799309260656640}]
I20260924 07:16:46.381461 23919 mvcc.cc:205] Tried to move back new op lower bound from 7332799309260656640 to 7332799309172936704. Current Snapshot: MvccSnapshot[applied={T|T < 7332799309260656640 or (T in {7332799309260656640})}]
I20260924 07:16:46.383647 23391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:46.383831 23391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:46.386627 23391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Sending DeleteTablet for 3 replicas of tablet 1f9b5e1de02b4dffb1a54a1ced3143c5
I20260924 07:16:46.387589 23529 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:46054
I20260924 07:16:46.387594 23809 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:38750
I20260924 07:16:46.388362 23669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42488
I20260924 07:16:46.388350 23391 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:46.388808 23391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:46.388960 23929 tablet_replica.cc:330] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: stopping tablet replica
I20260924 07:16:46.389302 23927 tablet_replica.cc:330] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: stopping tablet replica
I20260924 07:16:46.389544 23929 raft_consensus.cc:2229] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:46.389904 23927 raft_consensus.cc:2229] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:46.390002 23929 raft_consensus.cc:2258] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.390393 23927 raft_consensus.cc:2258] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.390869 23928 tablet_replica.cc:330] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: stopping tablet replica
I20260924 07:16:46.391238 23928 raft_consensus.cc:2229] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:46.391741 23928 raft_consensus.cc:2258] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.392356 23929 ts_tablet_manager.cc:1905] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.394518 23928 ts_tablet_manager.cc:1905] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.395346 23669 tablet_service.cc:1467] Processing CreateTablet for tablet b295066814214040ae01319c591893e9 (DEFAULT_TABLE table=test-table [id=ded478b938a34491a697880fbdd6129d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.395531 23929 ts_tablet_manager.cc:1918] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:46.395619 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b295066814214040ae01319c591893e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.395653 23929 log.cc:1196] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/1f9b5e1de02b4dffb1a54a1ced3143c5
I20260924 07:16:46.396014 23929 ts_tablet_manager.cc:1939] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 035c2d6231b141109fee34e97cde22d4: Deleting consensus metadata
I20260924 07:16:46.397910 23903 tablet_bootstrap.cc:492] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Bootstrap starting.
I20260924 07:16:46.398351 23809 tablet_service.cc:1467] Processing CreateTablet for tablet b295066814214040ae01319c591893e9 (DEFAULT_TABLE table=test-table [id=ded478b938a34491a697880fbdd6129d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.398686 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b295066814214040ae01319c591893e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.399245 23903 tablet_bootstrap.cc:654] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.400451 23903 tablet_bootstrap.cc:492] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: No bootstrap required, opened a new log
I20260924 07:16:46.400522 23903 ts_tablet_manager.cc:1397] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:46.400972 23903 raft_consensus.cc:348] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.401136 23903 raft_consensus.cc:374] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.401170 23903 raft_consensus.cc:729] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.401289 23903 consensus_queue.cc:260] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.401417 23376 catalog_manager.cc:4841] TS 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951): tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 (table test-table [id=1765d9dfc8f942b1833efbafe04c89a6]) successfully deleted
I20260924 07:16:46.401566 23903 ts_tablet_manager.cc:1428] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:46.401785 23905 tablet_bootstrap.cc:492] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Bootstrap starting.
I20260924 07:16:46.402240 23529 tablet_service.cc:1467] Processing CreateTablet for tablet b295066814214040ae01319c591893e9 (DEFAULT_TABLE table=test-table [id=ded478b938a34491a697880fbdd6129d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.402513 23529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b295066814214040ae01319c591893e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.402849 23905 tablet_bootstrap.cc:654] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.404860 23904 tablet_bootstrap.cc:492] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Bootstrap starting.
I20260924 07:16:46.406090 23904 tablet_bootstrap.cc:654] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.406236 23905 tablet_bootstrap.cc:492] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: No bootstrap required, opened a new log
I20260924 07:16:46.406325 23905 ts_tablet_manager.cc:1397] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:16:46.406827 23905 raft_consensus.cc:348] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.406955 23905 raft_consensus.cc:374] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.406996 23905 raft_consensus.cc:729] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801e3134ed4048908ed3dc3acc3ddcb9, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.407151 23905 consensus_queue.cc:260] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.407430 23905 ts_tablet_manager.cc:1428] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 07:16:46.407657 23927 ts_tablet_manager.cc:1905] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.411764 23927 ts_tablet_manager.cc:1918] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:46.411888 23927 log.cc:1196] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/1f9b5e1de02b4dffb1a54a1ced3143c5
I20260924 07:16:46.412201 23927 ts_tablet_manager.cc:1939] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting consensus metadata
I20260924 07:16:46.414788 23376 catalog_manager.cc:4841] TS 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321): tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 (table test-table [id=1765d9dfc8f942b1833efbafe04c89a6]) successfully deleted
I20260924 07:16:46.415947 23928 ts_tablet_manager.cc:1918] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:46.416095 23928 log.cc:1196] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/1f9b5e1de02b4dffb1a54a1ced3143c5
I20260924 07:16:46.416517 23928 ts_tablet_manager.cc:1939] T 1f9b5e1de02b4dffb1a54a1ced3143c5 P aca2034c82a14ce6aa8184b72e860316: Deleting consensus metadata
I20260924 07:16:46.418246 23904 tablet_bootstrap.cc:492] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: No bootstrap required, opened a new log
I20260924 07:16:46.418419 23904 ts_tablet_manager.cc:1397] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20260924 07:16:46.419421 23904 raft_consensus.cc:348] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.419636 23904 raft_consensus.cc:374] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.419703 23904 raft_consensus.cc:729] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.419945 23904 consensus_queue.cc:260] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.420421 23904 ts_tablet_manager.cc:1428] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:46.422195 23379 catalog_manager.cc:4841] TS aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449): tablet 1f9b5e1de02b4dffb1a54a1ced3143c5 (table test-table [id=1765d9dfc8f942b1833efbafe04c89a6]) successfully deleted
W20260924 07:16:46.432654 23876 tablet.cc:2369] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:46.447774 23736 tablet.cc:2369] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:16:46.460292 23596 tablet.cc:2369] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:16:46.504230 23911 raft_consensus.cc:482] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:46.504413 23911 raft_consensus.cc:504] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.505710 23911 leader_election.cc:288] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449), 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951)
I20260924 07:16:46.510669 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.506341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:36761 (local address 127.15.127.129:43449)
0924 07:16:46.506464 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.506469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.506485 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:46.506542 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.506545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.506589 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.506695 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.507051 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.507709 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.509357 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.509997 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.510148 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.510205 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.510445 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.510524 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:46.510669 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.505992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43449 (local address 127.15.127.131:36761)
0924 07:16:46.506174 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.506194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.506335 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.506728 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.506735 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.506756 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.506911 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.506917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.507838 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.507844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.509210 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.509222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.510148 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.510156 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.510287 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.510312 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.510422 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.510506 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 07:16:46.510812 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.506126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:42339 (local address 127.15.127.130:40951)
0924 07:16:46.506258 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.506266 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.506707 (+ 441us) server_negotiation.cc:407] Connection header received
0924 07:16:46.506892 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.506897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.506943 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.507018 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.507331 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.508131 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.509657 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.510127 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.510256 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.510312 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.510586 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.510679 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:16:46.511176 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.506009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:40951 (local address 127.15.127.131:42339)
0924 07:16:46.506595 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.506613 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.506766 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.507073 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.507076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.507095 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.507211 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.507217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.508267 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.508271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.509533 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.509541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.510301 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.510306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.510436 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.510460 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.510963 (+ 503us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.511038 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":283,"thread_start_us":80,"threads_started":1}
I20260924 07:16:46.511687 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295066814214040ae01319c591893e9" candidate_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035c2d6231b141109fee34e97cde22d4" is_pre_election: true
I20260924 07:16:46.511691 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295066814214040ae01319c591893e9" candidate_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca2034c82a14ce6aa8184b72e860316" is_pre_election: true
I20260924 07:16:46.511871 23689 raft_consensus.cc:2454] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801e3134ed4048908ed3dc3acc3ddcb9 in term 0.
I20260924 07:16:46.511938 23549 raft_consensus.cc:2454] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801e3134ed4048908ed3dc3acc3ddcb9 in term 0.
I20260924 07:16:46.512336 23760 leader_election.cc:302] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, 801e3134ed4048908ed3dc3acc3ddcb9, aca2034c82a14ce6aa8184b72e860316; no voters:
I20260924 07:16:46.512697 23911 raft_consensus.cc:2781] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:46.512778 23911 raft_consensus.cc:482] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:46.512825 23911 raft_consensus.cc:3037] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.514254 23911 raft_consensus.cc:504] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.514693 23911 leader_election.cc:288] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [CANDIDATE]: Term 1 election: Requested vote from peers aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449), 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951)
I20260924 07:16:46.515077 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295066814214040ae01319c591893e9" candidate_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035c2d6231b141109fee34e97cde22d4"
I20260924 07:16:46.515223 23689 raft_consensus.cc:3037] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.515430 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b295066814214040ae01319c591893e9" candidate_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca2034c82a14ce6aa8184b72e860316"
I20260924 07:16:46.515579 23549 raft_consensus.cc:3037] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.516265 23689 raft_consensus.cc:2454] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801e3134ed4048908ed3dc3acc3ddcb9 in term 1.
I20260924 07:16:46.516589 23760 leader_election.cc:302] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, 801e3134ed4048908ed3dc3acc3ddcb9; no voters:
I20260924 07:16:46.516788 23911 raft_consensus.cc:2781] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:46.516883 23549 raft_consensus.cc:2454] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801e3134ed4048908ed3dc3acc3ddcb9 in term 1.
I20260924 07:16:46.517428 23911 raft_consensus.cc:686] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 LEADER]: Becoming Leader. State: Replica: 801e3134ed4048908ed3dc3acc3ddcb9, State: Running, Role: LEADER
I20260924 07:16:46.517719 23911 consensus_queue.cc:237] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.520185 23391 catalog_manager.cc:5495] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131). New cstate: current_term: 1 leader_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } health_report { overall_health: HEALTHY } } }
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234205, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.190@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:46.568797 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.559052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46334 (local address 127.15.127.190:36169)
0924 07:16:46.559215 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.559221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.559396 (+ 175us) server_negotiation.cc:407] Connection header received
0924 07:16:46.559577 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.559581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.559629 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.559735 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.560492 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.561197 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.562149 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.562403 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.562524 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.565180 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:46.565184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:46.565187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:46.565216 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:46.566792 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.567380 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.567385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.567387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.567500 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.567783 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.567786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.567788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.568001 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:46.568147 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.568479 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.568614 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:16:46.568830 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.558918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36169 (local address 127.0.0.1:46334)
0924 07:16:46.559280 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.559294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.559458 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.559768 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.559770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.560253 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:46.560350 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.560357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.561329 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.561333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.561996 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.562006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.562531 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.562534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.562624 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.563348 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:46.563362 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:46.564961 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:46.566944 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:46.566957 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:46.566965 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:46.567280 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:46.567598 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:46.567601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:46.567603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:46.567690 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:46.568146 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:46.568155 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:46.568287 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.568434 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.568685 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":83,"thread_start_us":114,"threads_started":1}
I20260924 07:16:46.573581 23391 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:46334:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:16:46.574319 23391 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:46.579391 23809 tablet_service.cc:1467] Processing CreateTablet for tablet 7b257ae47282487c97c9d32674a3b1e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eb70289f18404f66912bc7b3383b7d65]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:46.579391 23529 tablet_service.cc:1467] Processing CreateTablet for tablet 7b257ae47282487c97c9d32674a3b1e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eb70289f18404f66912bc7b3383b7d65]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:46.579855 23529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b257ae47282487c97c9d32674a3b1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.579855 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b257ae47282487c97c9d32674a3b1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.581435 23669 tablet_service.cc:1467] Processing CreateTablet for tablet 7b257ae47282487c97c9d32674a3b1e8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eb70289f18404f66912bc7b3383b7d65]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:16:46.581911 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b257ae47282487c97c9d32674a3b1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.583079 23905 tablet_bootstrap.cc:492] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: Bootstrap starting.
I20260924 07:16:46.583106 23904 tablet_bootstrap.cc:492] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: Bootstrap starting.
I20260924 07:16:46.584023 23905 tablet_bootstrap.cc:654] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.584108 23904 tablet_bootstrap.cc:654] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.585264 23903 tablet_bootstrap.cc:492] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: Bootstrap starting.
I20260924 07:16:46.585397 23905 tablet_bootstrap.cc:492] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: No bootstrap required, opened a new log
I20260924 07:16:46.585490 23905 ts_tablet_manager.cc:1397] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:16:46.586359 23903 tablet_bootstrap.cc:654] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.586241 23905 raft_consensus.cc:348] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.586436 23905 raft_consensus.cc:374] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.586486 23905 raft_consensus.cc:729] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801e3134ed4048908ed3dc3acc3ddcb9, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.586633 23905 consensus_queue.cc:260] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.586699 23904 tablet_bootstrap.cc:492] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: No bootstrap required, opened a new log
I20260924 07:16:46.586796 23904 ts_tablet_manager.cc:1397] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:16:46.586915 23911 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.587054 23911 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.587361 23905 ts_tablet_manager.cc:1428] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:16:46.587348 23904 raft_consensus.cc:348] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.587471 23904 raft_consensus.cc:374] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.587523 23904 raft_consensus.cc:729] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.587699 23904 consensus_queue.cc:260] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.587947 23903 tablet_bootstrap.cc:492] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: No bootstrap required, opened a new log
I20260924 07:16:46.588048 23903 ts_tablet_manager.cc:1397] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:46.588199 23910 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.588356 23910 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.588588 23903 raft_consensus.cc:348] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.588712 23903 raft_consensus.cc:374] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.588753 23903 raft_consensus.cc:729] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.588889 23903 consensus_queue.cc:260] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.589215 23903 ts_tablet_manager.cc:1428] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:46.589308 23904 ts_tablet_manager.cc:1428] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:46.589437 23909 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.589587 23909 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.625628 23909 raft_consensus.cc:482] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:46.625773 23909 raft_consensus.cc:504] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.626978 23909 leader_election.cc:288] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.631261 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.627418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:39119 (local address 127.15.127.131:45321)
0924 07:16:46.627573 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.627578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.627705 (+ 127us) server_negotiation.cc:407] Connection header received
0924 07:16:46.627890 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.627896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.627952 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.628018 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.628366 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.628905 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.630228 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.630613 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.630795 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.630850 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.631026 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.631093 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:16:46.631290 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.627302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:45321 (local address 127.15.127.130:39119)
0924 07:16:46.627580 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.627599 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.627763 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.628064 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.628067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.628084 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.628240 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.628248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.629055 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.629059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.630072 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.630081 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.630795 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.630799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.630893 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.630915 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.631009 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.631086 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":35,"thread_start_us":83,"threads_started":1}
I20260924 07:16:46.631508 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.627792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:58585 (local address 127.15.127.129:43449)
0924 07:16:46.627925 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.627932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.627952 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:16:46.628048 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.628052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.628105 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.628196 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:16:46.628483 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.629043 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.630190 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.630656 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.630817 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.630881 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.631226 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.631315 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:16:46.631795 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b257ae47282487c97c9d32674a3b1e8" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" is_pre_election: true
I20260924 07:16:46.631942 23946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.627324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43449 (local address 127.15.127.130:58585)
0924 07:16:46.627714 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.627730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.627844 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.628221 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.628223 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.628237 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:16:46.628358 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.628364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.629164 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.629168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.630057 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.630066 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.630946 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.630951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.631071 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.631097 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.631661 (+ 564us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.631762 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":41,"thread_start_us":124,"threads_started":1}
I20260924 07:16:46.631963 23829 raft_consensus.cc:2454] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 0.
I20260924 07:16:46.632365 23620 leader_election.cc:302] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, 801e3134ed4048908ed3dc3acc3ddcb9; no voters:
I20260924 07:16:46.632416 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b257ae47282487c97c9d32674a3b1e8" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca2034c82a14ce6aa8184b72e860316" is_pre_election: true
I20260924 07:16:46.632586 23549 raft_consensus.cc:2454] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 0.
I20260924 07:16:46.632616 23909 raft_consensus.cc:2781] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:46.632675 23909 raft_consensus.cc:482] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:46.632755 23909 raft_consensus.cc:3037] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.633843 23909 raft_consensus.cc:504] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.634229 23909 leader_election.cc:288] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 election: Requested vote from peers aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.634608 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b257ae47282487c97c9d32674a3b1e8" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
I20260924 07:16:46.634763 23829 raft_consensus.cc:3037] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.635839 23829 raft_consensus.cc:2454] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 1.
I20260924 07:16:46.636114 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b257ae47282487c97c9d32674a3b1e8" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca2034c82a14ce6aa8184b72e860316"
I20260924 07:16:46.636215 23620 leader_election.cc:302] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, 801e3134ed4048908ed3dc3acc3ddcb9; no voters:
I20260924 07:16:46.636268 23549 raft_consensus.cc:3037] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.636443 23909 raft_consensus.cc:2781] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:46.636660 23909 raft_consensus.cc:686] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [term 1 LEADER]: Becoming Leader. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Running, Role: LEADER
I20260924 07:16:46.636929 23909 consensus_queue.cc:237] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.637411 23549 raft_consensus.cc:2454] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 1.
I20260924 07:16:46.638454 23947 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: TxnStatusTablet state changed. Reason: New leader 035c2d6231b141109fee34e97cde22d4. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.638569 23947 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:46.638859 23949 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:16:46.639035 23949 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:16:46.639539 23392 catalog_manager.cc:5495] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 035c2d6231b141109fee34e97cde22d4 (127.15.127.130). New cstate: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } health_report { overall_health: UNKNOWN } } }
I20260924 07:16:46.655485 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.651024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.15.127.130:40951)
0924 07:16:46.651229 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.651232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.651247 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:46.651377 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.651379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.651434 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.651523 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:46.651879 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.652495 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.653796 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.654043 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.654138 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.654588 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.655051 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.655168 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.655311 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.655369 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260924 07:16:46.655520 23941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.650886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:40951 (local address 127.0.0.1:42492)
0924 07:16:46.651088 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.651110 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.651230 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.651564 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.651567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.651576 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:46.651756 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.651764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.652612 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.652617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.653654 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.653666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.654184 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.654188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.654293 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.654360 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.655162 (+ 802us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.655182 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.655260 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.655367 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 07:16:46.659024 23829 raft_consensus.cc:1261] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Refusing update from remote peer 035c2d6231b141109fee34e97cde22d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.659191 23549 raft_consensus.cc:1261] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Refusing update from remote peer 035c2d6231b141109fee34e97cde22d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.659588 23947 consensus_queue.cc:1035] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.659782 23909 consensus_queue.cc:1035] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.661384 23911 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: TxnStatusTablet state changed. Reason: New leader 035c2d6231b141109fee34e97cde22d4. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.661511 23911 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.662858 23910 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: TxnStatusTablet state changed. Reason: New leader 035c2d6231b141109fee34e97cde22d4. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.662997 23910 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.664243 23909 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.664366 23909 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:46.665400 23911 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.665517 23911 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 801e3134ed4048908ed3dc3acc3ddcb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.666180 23918 mvcc.cc:205] Tried to move back new op lower bound from 7332799310469742592 to 7332799310387654656. Current Snapshot: MvccSnapshot[applied={T|T < 7332799310469742592 or (T in {7332799310469742592})}]
I20260924 07:16:46.668651 23909 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.668808 23909 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P 035c2d6231b141109fee34e97cde22d4: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:16:46.670176 23910 tablet_replica.cc:439] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } } }
I20260924 07:16:46.670312 23910 tablet_replica.cc:442] T 7b257ae47282487c97c9d32674a3b1e8 P aca2034c82a14ce6aa8184b72e860316: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:16:46.674533 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.669594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.15.127.131:45321)
0924 07:16:46.669726 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.669731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.669746 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:46.669827 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.669830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.669896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.670006 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:46.671000 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.671785 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.673046 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.673307 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.673391 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.673687 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.674168 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.674256 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.674354 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.674405 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:16:46.685359 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.681857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:40951 (local address 127.15.127.131:44417)
0924 07:16:46.682031 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.682072 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.682173 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.682408 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.682411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.682419 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:46.682559 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.682565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.683300 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.683305 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.684122 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.684132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.684587 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.684591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.684689 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.684736 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.685074 (+ 338us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.685081 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.685145 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.685235 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 07:16:46.685359 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.682019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:44417 (local address 127.15.127.130:40951)
0924 07:16:46.682161 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.682165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.682181 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:46.682266 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.682269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.682319 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.682399 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:46.682689 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.683192 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.684245 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.684467 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.684586 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.684879 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.684955 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.685060 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.685153 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.685205 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:16:46.693564 23549 raft_consensus.cc:1261] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Refusing update from remote peer 801e3134ed4048908ed3dc3acc3ddcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.693728 23689 raft_consensus.cc:1261] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Refusing update from remote peer 801e3134ed4048908ed3dc3acc3ddcb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.694139 23934 consensus_queue.cc:1035] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.694347 23911 consensus_queue.cc:1035] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.725620 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.720206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:47277 (local address 127.15.127.131:45321)
0924 07:16:46.720327 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.720332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.721309 (+ 977us) server_negotiation.cc:407] Connection header received
0924 07:16:46.721527 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.721531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.721613 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.721664 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:46.722182 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.723025 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.724166 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.724503 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.724583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.725094 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.725191 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.725238 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.725415 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.725478 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:16:46.726197 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.720140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:45321 (local address 127.15.127.130:47277)
0924 07:16:46.721223 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:46.721241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:46.721398 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:46.721863 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:46.721866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:46.721876 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:46.722004 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.722010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.723140 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.723142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.724007 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:46.724025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.724738 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.724742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:46.724846 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.724909 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.725305 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.725314 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:16:46.726003 (+ 689us) client_negotiation.cc:240] Negotiation successful
0924 07:16:46.726078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"thread_start_us":105,"threads_started":1}
I20260924 07:16:46.756807 23392 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:46.757040 23392 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:46.759698 23392 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Sending DeleteTablet for 3 replicas of tablet b295066814214040ae01319c591893e9
I20260924 07:16:46.760864 23669 tablet_service.cc:1514] Processing DeleteTablet for tablet b295066814214040ae01319c591893e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42488
I20260924 07:16:46.760881 23809 tablet_service.cc:1514] Processing DeleteTablet for tablet b295066814214040ae01319c591893e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:38750
I20260924 07:16:46.761122 23929 tablet_replica.cc:330] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: stopping tablet replica
I20260924 07:16:46.761260 23929 raft_consensus.cc:2229] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:46.761386 23929 raft_consensus.cc:2258] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.761689 23529 tablet_service.cc:1514] Processing DeleteTablet for tablet b295066814214040ae01319c591893e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:46054
I20260924 07:16:46.761940 23928 tablet_replica.cc:330] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: stopping tablet replica
I20260924 07:16:46.762125 23928 raft_consensus.cc:2229] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:46.762269 23928 raft_consensus.cc:2258] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.762756 23929 ts_tablet_manager.cc:1905] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.763181 23928 ts_tablet_manager.cc:1905] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.765974 23929 ts_tablet_manager.cc:1918] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:46.766042 23928 ts_tablet_manager.cc:1918] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:46.766080 23929 log.cc:1196] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/b295066814214040ae01319c591893e9
I20260924 07:16:46.766131 23928 log.cc:1196] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/b295066814214040ae01319c591893e9
I20260924 07:16:46.766374 23929 ts_tablet_manager.cc:1939] T b295066814214040ae01319c591893e9 P 035c2d6231b141109fee34e97cde22d4: Deleting consensus metadata
I20260924 07:16:46.766397 23928 ts_tablet_manager.cc:1939] T b295066814214040ae01319c591893e9 P aca2034c82a14ce6aa8184b72e860316: Deleting consensus metadata
I20260924 07:16:46.767243 23379 catalog_manager.cc:4841] TS aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449): tablet b295066814214040ae01319c591893e9 (table test-table [id=ded478b938a34491a697880fbdd6129d]) successfully deleted
I20260924 07:16:46.767376 23376 catalog_manager.cc:4841] TS 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951): tablet b295066814214040ae01319c591893e9 (table test-table [id=ded478b938a34491a697880fbdd6129d]) successfully deleted
I20260924 07:16:46.767699 23927 tablet_replica.cc:330] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: stopping tablet replica
I20260924 07:16:46.767864 23927 raft_consensus.cc:2229] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:46.768286 23927 raft_consensus.cc:2258] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.770702 23927 ts_tablet_manager.cc:1905] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.773883 23927 ts_tablet_manager.cc:1918] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:46.773962 23927 log.cc:1196] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/b295066814214040ae01319c591893e9
I20260924 07:16:46.774264 23927 ts_tablet_manager.cc:1939] T b295066814214040ae01319c591893e9 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting consensus metadata
I20260924 07:16:46.775182 23376 catalog_manager.cc:4841] TS 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321): tablet b295066814214040ae01319c591893e9 (table test-table [id=ded478b938a34491a697880fbdd6129d]) successfully deleted
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:16:46.797941 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:46.791167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.15.127.190:36169)
0924 07:16:46.791314 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.791319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.791581 (+ 262us) server_negotiation.cc:407] Connection header received
0924 07:16:46.791754 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.791760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.791812 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.791906 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.792663 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.793290 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.794164 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.794382 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.794501 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.797797 (+ 3296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46348: BlockingRecv error: recv got EOF from 127.0.0.1:46348 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 07:16:46 dist-test-slave-zx46 krb5kdc[23347](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM
I20260924 07:16:46.808761 23441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.800219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.15.127.190:36169)
0924 07:16:46.800335 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.800343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.800378 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:16:46.800549 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.800553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.800604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.800690 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:46.801375 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.801895 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.802702 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.802926 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.803040 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.805495 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:46.805499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:46.805501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:46.805530 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:46.807010 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.807461 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.807465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.807467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.807564 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:46.807849 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:46.807853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:46.807855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:46.808051 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:46.808194 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.808439 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.808571 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:16:46.811789 23392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:46.812386 23392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:46.817772 23529 tablet_service.cc:1467] Processing CreateTablet for tablet c41686fd75e347f68b869f37a6396efc (DEFAULT_TABLE table=test-table [id=38e92de4f99c49d3abe758e6c8f76729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.818130 23529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41686fd75e347f68b869f37a6396efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.818077 23809 tablet_service.cc:1467] Processing CreateTablet for tablet c41686fd75e347f68b869f37a6396efc (DEFAULT_TABLE table=test-table [id=38e92de4f99c49d3abe758e6c8f76729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.818331 23809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41686fd75e347f68b869f37a6396efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.820762 23904 tablet_bootstrap.cc:492] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Bootstrap starting.
I20260924 07:16:46.821727 23904 tablet_bootstrap.cc:654] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.822388 23905 tablet_bootstrap.cc:492] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Bootstrap starting.
I20260924 07:16:46.823416 23905 tablet_bootstrap.cc:654] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.824426 23669 tablet_service.cc:1467] Processing CreateTablet for tablet c41686fd75e347f68b869f37a6396efc (DEFAULT_TABLE table=test-table [id=38e92de4f99c49d3abe758e6c8f76729]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.824712 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c41686fd75e347f68b869f37a6396efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.826689 23903 tablet_bootstrap.cc:492] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Bootstrap starting.
I20260924 07:16:46.827687 23903 tablet_bootstrap.cc:654] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.828902 23904 tablet_bootstrap.cc:492] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: No bootstrap required, opened a new log
I20260924 07:16:46.829044 23904 ts_tablet_manager.cc:1397] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 07:16:46.829479 23904 raft_consensus.cc:348] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.829612 23904 raft_consensus.cc:374] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.829653 23904 raft_consensus.cc:729] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.829777 23904 consensus_queue.cc:260] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.830380 23905 tablet_bootstrap.cc:492] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: No bootstrap required, opened a new log
I20260924 07:16:46.830489 23905 ts_tablet_manager.cc:1397] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260924 07:16:46.830991 23905 raft_consensus.cc:348] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.831121 23905 raft_consensus.cc:374] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.831162 23905 raft_consensus.cc:729] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801e3134ed4048908ed3dc3acc3ddcb9, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.831314 23905 consensus_queue.cc:260] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.831615 23905 ts_tablet_manager.cc:1428] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:16:46.836164 23904 ts_tablet_manager.cc:1428] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260924 07:16:46.838019 23903 tablet_bootstrap.cc:492] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: No bootstrap required, opened a new log
I20260924 07:16:46.838171 23903 ts_tablet_manager.cc:1397] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260924 07:16:46.838991 23903 raft_consensus.cc:348] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.839174 23903 raft_consensus.cc:374] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.839221 23903 raft_consensus.cc:729] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.839673 23903 consensus_queue.cc:260] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.840070 23903 ts_tablet_manager.cc:1428] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:16:46.878765 23947 raft_consensus.cc:482] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:46.878924 23947 raft_consensus.cc:504] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.879503 23947 leader_election.cc:288] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.879992 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41686fd75e347f68b869f37a6396efc" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" is_pre_election: true
I20260924 07:16:46.880030 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41686fd75e347f68b869f37a6396efc" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca2034c82a14ce6aa8184b72e860316" is_pre_election: true
I20260924 07:16:46.880189 23829 raft_consensus.cc:2454] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 0.
I20260924 07:16:46.880223 23549 raft_consensus.cc:2454] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 0.
I20260924 07:16:46.880604 23623 leader_election.cc:302] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, aca2034c82a14ce6aa8184b72e860316; no voters:
I20260924 07:16:46.880856 23947 raft_consensus.cc:2781] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:46.880929 23947 raft_consensus.cc:482] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:46.880992 23947 raft_consensus.cc:3037] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.882149 23947 raft_consensus.cc:504] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.882632 23947 leader_election.cc:288] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 election: Requested vote from peers aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.883068 23549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41686fd75e347f68b869f37a6396efc" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aca2034c82a14ce6aa8184b72e860316"
I20260924 07:16:46.883134 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c41686fd75e347f68b869f37a6396efc" candidate_uuid: "035c2d6231b141109fee34e97cde22d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
I20260924 07:16:46.883226 23549 raft_consensus.cc:3037] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.883268 23829 raft_consensus.cc:3037] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.884276 23829 raft_consensus.cc:2454] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 1.
I20260924 07:16:46.884543 23549 raft_consensus.cc:2454] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035c2d6231b141109fee34e97cde22d4 in term 1.
I20260924 07:16:46.884661 23620 leader_election.cc:302] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, 801e3134ed4048908ed3dc3acc3ddcb9; no voters:
I20260924 07:16:46.884873 23947 raft_consensus.cc:2781] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:46.885063 23947 raft_consensus.cc:686] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 1 LEADER]: Becoming Leader. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Running, Role: LEADER
I20260924 07:16:46.885214 23947 consensus_queue.cc:237] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.887064 23391 catalog_manager.cc:5495] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 035c2d6231b141109fee34e97cde22d4 (127.15.127.130). New cstate: current_term: 1 leader_uuid: "035c2d6231b141109fee34e97cde22d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } health_report { overall_health: UNKNOWN } } }
I20260924 07:16:46.904937 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:46.900888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42500 (local address 127.15.127.130:40951)
0924 07:16:46.901060 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:46.901074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:46.901088 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:46.901230 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:46.901232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:46.901275 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:46.901345 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:46.901709 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.902370 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.903650 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:46.903892 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:46.903988 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:46.904318 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:46.904396 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:46.904454 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:46.904656 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 07:16:46.904719 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:16:46.907699 23829 raft_consensus.cc:1261] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Refusing update from remote peer 035c2d6231b141109fee34e97cde22d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.907833 23549 raft_consensus.cc:1261] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Refusing update from remote peer 035c2d6231b141109fee34e97cde22d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:46.908389 23909 consensus_queue.cc:1035] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.908604 23947 consensus_queue.cc:1035] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:46.919804 23391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:46.919998 23391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:46.923079 23391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Sending DeleteTablet for 3 replicas of tablet c41686fd75e347f68b869f37a6396efc
I20260924 07:16:46.924408 23809 tablet_service.cc:1514] Processing DeleteTablet for tablet c41686fd75e347f68b869f37a6396efc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:38750
I20260924 07:16:46.924499 23669 tablet_service.cc:1514] Processing DeleteTablet for tablet c41686fd75e347f68b869f37a6396efc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42488
I20260924 07:16:46.924690 23929 tablet_replica.cc:330] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: stopping tablet replica
I20260924 07:16:46.924724 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet c41686fd75e347f68b869f37a6396efc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:38750
I20260924 07:16:46.924830 23929 raft_consensus.cc:2229] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:46.924865 23927 tablet_replica.cc:330] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: stopping tablet replica
I20260924 07:16:46.924952 23927 raft_consensus.cc:2229] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:46.925225 23929 raft_consensus.cc:2258] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.925313 23927 raft_consensus.cc:2258] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.925889 23529 tablet_service.cc:1514] Processing DeleteTablet for tablet c41686fd75e347f68b869f37a6396efc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:46 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:46054
I20260924 07:16:46.926077 23928 tablet_replica.cc:330] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: stopping tablet replica
I20260924 07:16:46.926175 23929 ts_tablet_manager.cc:1905] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.926182 23928 raft_consensus.cc:2229] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:46.926306 23928 raft_consensus.cc:2258] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:46.927659 23392 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:16:46.928301 23392 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:16:46.928920 23929 ts_tablet_manager.cc:1918] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:46.929059 23929 log.cc:1196] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/c41686fd75e347f68b869f37a6396efc
I20260924 07:16:46.929332 23929 ts_tablet_manager.cc:1939] T c41686fd75e347f68b869f37a6396efc P 035c2d6231b141109fee34e97cde22d4: Deleting consensus metadata
I20260924 07:16:46.930085 23376 catalog_manager.cc:4841] TS 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951): tablet c41686fd75e347f68b869f37a6396efc (table test-table [id=38e92de4f99c49d3abe758e6c8f76729]) successfully deleted
I20260924 07:16:46.931133 23928 ts_tablet_manager.cc:1905] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.933734 23928 ts_tablet_manager.cc:1918] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:46.933826 23928 log.cc:1196] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/c41686fd75e347f68b869f37a6396efc
I20260924 07:16:46.934095 23928 ts_tablet_manager.cc:1939] T c41686fd75e347f68b869f37a6396efc P aca2034c82a14ce6aa8184b72e860316: Deleting consensus metadata
I20260924 07:16:46.935405 23927 ts_tablet_manager.cc:1905] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:46.937500 23927 ts_tablet_manager.cc:1918] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:16:46.937618 23927 log.cc:1196] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/c41686fd75e347f68b869f37a6396efc
I20260924 07:16:46.937855 23927 ts_tablet_manager.cc:1939] T c41686fd75e347f68b869f37a6396efc P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting consensus metadata
I20260924 07:16:46.938656 23376 catalog_manager.cc:4841] TS 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321): tablet c41686fd75e347f68b869f37a6396efc (table test-table [id=38e92de4f99c49d3abe758e6c8f76729]) successfully deleted
W20260924 07:16:46.939028 23376 catalog_manager.cc:4808] TS 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321): delete failed for tablet c41686fd75e347f68b869f37a6396efc because the tablet was not found. No further retry: Not found: Tablet not found: c41686fd75e347f68b869f37a6396efc
I20260924 07:16:46.940333 23529 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc92469719846abbd973e03d505e4f8 (DEFAULT_TABLE table=test-table [id=c67e8926c75348a09611e2501e619b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.940690 23529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc92469719846abbd973e03d505e4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.943143 23904 tablet_bootstrap.cc:492] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: Bootstrap starting.
I20260924 07:16:46.943423 23808 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc92469719846abbd973e03d505e4f8 (DEFAULT_TABLE table=test-table [id=c67e8926c75348a09611e2501e619b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.943728 23808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc92469719846abbd973e03d505e4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.944165 23904 tablet_bootstrap.cc:654] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.945694 23904 tablet_bootstrap.cc:492] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: No bootstrap required, opened a new log
I20260924 07:16:46.945788 23904 ts_tablet_manager.cc:1397] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:46.946310 23905 tablet_bootstrap.cc:492] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Bootstrap starting.
I20260924 07:16:46.946254 23904 raft_consensus.cc:348] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.946373 23904 raft_consensus.cc:374] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.946404 23904 raft_consensus.cc:729] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.946539 23904 consensus_queue.cc:260] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.946827 23904 ts_tablet_manager.cc:1428] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:16:46.947312 23905 tablet_bootstrap.cc:654] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.949142 23669 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc92469719846abbd973e03d505e4f8 (DEFAULT_TABLE table=test-table [id=c67e8926c75348a09611e2501e619b63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:16:46.949424 23669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc92469719846abbd973e03d505e4f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:46.951502 23905 tablet_bootstrap.cc:492] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: No bootstrap required, opened a new log
I20260924 07:16:46.951608 23905 ts_tablet_manager.cc:1397] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260924 07:16:46.952171 23903 tablet_bootstrap.cc:492] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Bootstrap starting.
I20260924 07:16:46.952103 23905 raft_consensus.cc:348] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.952229 23905 raft_consensus.cc:374] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.952260 23905 raft_consensus.cc:729] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801e3134ed4048908ed3dc3acc3ddcb9, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.953336 23903 tablet_bootstrap.cc:654] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:46.954896 23905 consensus_queue.cc:260] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.955363 23905 ts_tablet_manager.cc:1428] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260924 07:16:46.956462 23379 catalog_manager.cc:4841] TS aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449): tablet c41686fd75e347f68b869f37a6396efc (table test-table [id=38e92de4f99c49d3abe758e6c8f76729]) successfully deleted
I20260924 07:16:46.958729 23903 tablet_bootstrap.cc:492] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: No bootstrap required, opened a new log
I20260924 07:16:46.958840 23903 ts_tablet_manager.cc:1397] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260924 07:16:46.959350 23903 raft_consensus.cc:348] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.959481 23903 raft_consensus.cc:374] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:46.959524 23903 raft_consensus.cc:729] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035c2d6231b141109fee34e97cde22d4, State: Initialized, Role: FOLLOWER
I20260924 07:16:46.959658 23903 consensus_queue.cc:260] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.960053 23903 ts_tablet_manager.cc:1428] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 07:16:46.995992 23910 raft_consensus.cc:482] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:16:46.996212 23910 raft_consensus.cc:504] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.996703 23910 leader_election.cc:288] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:46.997231 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc92469719846abbd973e03d505e4f8" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035c2d6231b141109fee34e97cde22d4" is_pre_election: true
I20260924 07:16:46.997253 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc92469719846abbd973e03d505e4f8" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" is_pre_election: true
I20260924 07:16:46.997404 23689 raft_consensus.cc:2454] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 0.
I20260924 07:16:46.997421 23829 raft_consensus.cc:2454] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 0.
I20260924 07:16:46.997756 23480 leader_election.cc:302] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801e3134ed4048908ed3dc3acc3ddcb9, aca2034c82a14ce6aa8184b72e860316; no voters:
I20260924 07:16:46.997951 23910 raft_consensus.cc:2781] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:16:46.998023 23910 raft_consensus.cc:482] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:16:46.998076 23910 raft_consensus.cc:3037] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:46.999375 23910 raft_consensus.cc:504] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:46.999814 23910 leader_election.cc:288] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 election: Requested vote from peers 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951), 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321)
I20260924 07:16:47.000309 23689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc92469719846abbd973e03d505e4f8" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035c2d6231b141109fee34e97cde22d4"
I20260924 07:16:47.000440 23689 raft_consensus.cc:3037] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:47.000442 23829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc92469719846abbd973e03d505e4f8" candidate_uuid: "aca2034c82a14ce6aa8184b72e860316" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801e3134ed4048908ed3dc3acc3ddcb9"
I20260924 07:16:47.000584 23829 raft_consensus.cc:3037] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:47.001673 23689 raft_consensus.cc:2454] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 1.
I20260924 07:16:47.001685 23829 raft_consensus.cc:2454] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aca2034c82a14ce6aa8184b72e860316 in term 1.
I20260924 07:16:47.002005 23480 leader_election.cc:302] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035c2d6231b141109fee34e97cde22d4, aca2034c82a14ce6aa8184b72e860316; no voters:
I20260924 07:16:47.002223 23910 raft_consensus.cc:2781] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:47.002396 23910 raft_consensus.cc:686] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 1 LEADER]: Becoming Leader. State: Replica: aca2034c82a14ce6aa8184b72e860316, State: Running, Role: LEADER
I20260924 07:16:47.002765 23910 consensus_queue.cc:237] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } }
I20260924 07:16:47.004765 23391 catalog_manager.cc:5495] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 reported cstate change: term changed from 0 to 1, leader changed from <none> to aca2034c82a14ce6aa8184b72e860316 (127.15.127.129). New cstate: current_term: 1 leader_uuid: "aca2034c82a14ce6aa8184b72e860316" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aca2034c82a14ce6aa8184b72e860316" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 } health_report { overall_health: UNKNOWN } } }
I20260924 07:16:47.031819 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:47.027678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46068 (local address 127.15.127.129:43449)
0924 07:16:47.027790 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:47.027794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:47.027814 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:16:47.027862 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:47.027864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:47.027904 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:47.027995 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:47.028274 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.029169 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.030527 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.030931 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.031038 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:47.031341 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:47.031433 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:47.031516 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:47.031614 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:16:47.031671 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:16:47.045300 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:47.041128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60629 (local address 127.15.127.130:40951)
0924 07:16:47.041244 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:47.041249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:47.041265 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:47.041326 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:47.041329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:47.041388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:47.041483 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:47.041775 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.042624 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.043868 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.044219 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.044296 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:47.044746 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:47.044836 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:47.044877 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:47.045104 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:16:47.045160 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:16:47.045699 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:47.040170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:40951 (local address 127.15.127.129:60629)
0924 07:16:47.040495 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:47.040512 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:47.040634 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:47.041519 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:47.041523 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:47.041533 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:47.041652 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:47.041658 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.042737 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.042741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:47.043722 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:47.043734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.044416 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.044419 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:47.044525 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:47.044587 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:47.044936 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:47.044941 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:16:47.045460 (+ 519us) client_negotiation.cc:240] Negotiation successful
0924 07:16:47.045555 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260924 07:16:47.052374 23829 raft_consensus.cc:1261] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Refusing update from remote peer aca2034c82a14ce6aa8184b72e860316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:47.052407 23689 raft_consensus.cc:1261] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Refusing update from remote peer aca2034c82a14ce6aa8184b72e860316: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:16:47.052863 23910 consensus_queue.cc:1035] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035c2d6231b141109fee34e97cde22d4" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 40951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:47.053112 23986 consensus_queue.cc:1035] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [LEADER]: Connected to new peer: Peer: permanent_uuid: "801e3134ed4048908ed3dc3acc3ddcb9" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 45321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:16:47.073601 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:47.070114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:59011 (local address 127.15.127.129:43449)
0924 07:16:47.070229 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:47.070234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:47.070277 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:16:47.070400 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:47.070404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:47.070457 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:47.070535 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:16:47.070782 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.071505 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.072402 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.072698 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.072827 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:47.073169 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:16:47.073242 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:16:47.073300 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:47.073432 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:16:47.073480 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:16:47.073664 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:47.069893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43449 (local address 127.15.127.130:59011)
0924 07:16:47.070164 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:47.070182 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:47.070267 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:47.070547 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:47.070550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:47.070566 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:16:47.070675 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:47.070680 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.071666 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.071670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:47.072279 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:47.072288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:47.072857 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:47.072861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:47.072956 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:47.072993 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:16:47.073338 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:16:47.073346 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 07:16:47.073427 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:16:47.073521 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260924 07:16:47.096616 23391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:47.096793 23391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:16:47.099449 23391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25f368d115a0422ca941549e2a942b93: Sending DeleteTablet for 3 replicas of tablet 1fc92469719846abbd973e03d505e4f8
I20260924 07:16:47.100224 23529 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc92469719846abbd973e03d505e4f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:47 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:46054
I20260924 07:16:47.100350 23808 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc92469719846abbd973e03d505e4f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:47 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:38750
I20260924 07:16:47.100365 23669 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc92469719846abbd973e03d505e4f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:16:47 UTC) from {username='oryx', principal='oryx/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:42488
I20260924 07:16:47.100420 23928 tablet_replica.cc:330] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: stopping tablet replica
I20260924 07:16:47.100539 23927 tablet_replica.cc:330] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: stopping tablet replica
I20260924 07:16:47.100539 23929 tablet_replica.cc:330] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: stopping tablet replica
I20260924 07:16:47.100530 23928 raft_consensus.cc:2229] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:16:47.100639 23929 raft_consensus.cc:2229] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:47.100651 23927 raft_consensus.cc:2229] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:16:47.100726 23928 raft_consensus.cc:2258] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:47.100766 23927 raft_consensus.cc:2258] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:47.100766 23929 raft_consensus.cc:2258] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:16:47.101573 23928 ts_tablet_manager.cc:1905] T 1fc92469719846abbd973e03d505e4f8 P aca2034c82a14ce6aa8184b72e860316: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:47.101577 23927 ts_tablet_manager.cc:1905] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:47.103183 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 23464
I20260924 07:16:47.104678 23927 ts_tablet_manager.cc:1918] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:47.104772 23927 log.cc:1196] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/1fc92469719846abbd973e03d505e4f8
I20260924 07:16:47.105096 23927 ts_tablet_manager.cc:1939] T 1fc92469719846abbd973e03d505e4f8 P 801e3134ed4048908ed3dc3acc3ddcb9: Deleting consensus metadata
I20260924 07:16:47.105926 23929 ts_tablet_manager.cc:1905] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:16:47.105897 23376 catalog_manager.cc:4841] TS 801e3134ed4048908ed3dc3acc3ddcb9 (127.15.127.131:45321): tablet 1fc92469719846abbd973e03d505e4f8 (table test-table [id=c67e8926c75348a09611e2501e619b63]) successfully deleted
I20260924 07:16:47.107963 23929 ts_tablet_manager.cc:1918] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:16:47.108034 23929 log.cc:1196] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/1fc92469719846abbd973e03d505e4f8
I20260924 07:16:47.108299 23929 ts_tablet_manager.cc:1939] T 1fc92469719846abbd973e03d505e4f8 P 035c2d6231b141109fee34e97cde22d4: Deleting consensus metadata
I20260924 07:16:47.109373 23376 catalog_manager.cc:4841] TS 035c2d6231b141109fee34e97cde22d4 (127.15.127.130:40951): tablet 1fc92469719846abbd973e03d505e4f8 (table test-table [id=c67e8926c75348a09611e2501e619b63]) successfully deleted
W20260924 07:16:47.113098 23379 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:43449 (error 108)
I20260924 07:16:47.114292 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:47.113925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43449 (local address 127.0.0.1:46070)
0924 07:16:47.114128 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:47.114200 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:43449: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260924 07:16:47.114483 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 23604
W20260924 07:16:47.117751 23379 catalog_manager.cc:4584] TS aca2034c82a14ce6aa8184b72e860316 (127.15.127.129:43449): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fc92469719846abbd973e03d505e4f8: Network error: Client connection negotiation failed: client connection to 127.15.127.129:43449: connect: Connection refused (error 111)
I20260924 07:16:47.122910 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 23744
I20260924 07:16:47.130066 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 23360
2026-09-24T07:16:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3110 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:47 dist-test-slave-zx46 krb5kdc[23998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:47 dist-test-slave-zx46 krb5kdc[23998](info): set up 2 sockets
Sep 24 07:16:47 dist-test-slave-zx46 krb5kdc[23998](info): commencing operation
krb5kdc: starting...
W20260924 07:16:48.190034 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:48 dist-test-slave-zx46 krb5kdc[23998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:48.322469 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:40371
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:40371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:48.442195 24011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:48.442507 24011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:48.442552 24011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:48.446928 24011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:48.447016 24011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:48.447044 24011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:48.447063 24011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:48.447104 24011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:48.453132 24011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:40371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:40371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:48.454399 24011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:48.455973 24011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:48.466099 24016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:48.466094 24017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:48.466221 24011 server_base.cc:1034] running on GCE node
W20260924 07:16:48.466094 24019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:48.467078 24011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:48.468395 24011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:48.469573 24011 hybrid_clock.cc:648] HybridClock initialized: now 1790234208469535 us; error 56 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T07:16:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1349 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:16:48 dist-test-slave-zx46 krb5kdc[24024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:16:48 dist-test-slave-zx46 krb5kdc[24024](info): set up 2 sockets
Sep 24 07:16:48 dist-test-slave-zx46 krb5kdc[24024](info): commencing operation
krb5kdc: starting...
W20260924 07:16:49.537772 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:16:49 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:16:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:16:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:16:49.668826 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:45483
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:49.784636 24037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:49.784992 24037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:49.785099 24037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:49.789350 24037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:49.789479 24037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:49.789517 24037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:49.789556 24037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:49.789597 24037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:49.795480 24037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:49.796757 24037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:49.797946 24037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:49.807464 24045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:49.807458 24042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:49.807502 24037 server_base.cc:1034] running on GCE node
W20260924 07:16:49.807461 24043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:49.808383 24037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:49.809676 24037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:49.810895 24037 hybrid_clock.cc:648] HybridClock initialized: now 1790234209810879 us; error 31 us; skew 500 ppm
Sep 24 07:16:49 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234209, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:49.819816 24037 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:49.820865 24037 webserver.cc:460] Webserver started at http://127.15.127.190:34777/ using document root <none> and password file <none>
I20260924 07:16:49.821573 24037 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:49.821671 24037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:49.821914 24037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:49.824133 24037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "cb485e706632411da22a0c8e60d79c7f"
format_stamp: "Formatted at 2026-09-24 07:16:49 on dist-test-slave-zx46"
server_key: "0b1e9f00b7a9ade1f78e1c6bc2929b68"
server_key_iv: "bbe57a278377ea232abd24df78b91118"
server_key_version: "encryptionkey@0"
I20260924 07:16:49.824745 24037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "cb485e706632411da22a0c8e60d79c7f"
format_stamp: "Formatted at 2026-09-24 07:16:49 on dist-test-slave-zx46"
server_key: "0b1e9f00b7a9ade1f78e1c6bc2929b68"
server_key_iv: "bbe57a278377ea232abd24df78b91118"
server_key_version: "encryptionkey@0"
I20260924 07:16:49.828751 24037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:16:49.831636 24052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:49.833089 24037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:49.833241 24037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "cb485e706632411da22a0c8e60d79c7f"
format_stamp: "Formatted at 2026-09-24 07:16:49 on dist-test-slave-zx46"
server_key: "0b1e9f00b7a9ade1f78e1c6bc2929b68"
server_key_iv: "bbe57a278377ea232abd24df78b91118"
server_key_version: "encryptionkey@0"
I20260924 07:16:49.833369 24037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:49.854841 24037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:49.858433 24037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:49.858695 24037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:49.866343 24037 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:45483
I20260924 07:16:49.866413 24104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:45483 every 8 connection(s)
I20260924 07:16:49.867347 24037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:49.870334 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:16:49.876278 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24037
I20260924 07:16:49.876366 24105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Bootstrap starting.
I20260924 07:16:49.876437 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:16:49.876751 15870 external_mini_cluster.cc:1428] Setting key 2134b52a9d8387cbdda43641e8b8b142
I20260924 07:16:49.879331 24105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Neither blocks nor log segments found. Creating new log.
I20260924 07:16:49.880165 24105 log.cc:824] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:49.882346 24105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: No bootstrap required, opened a new log
Sep 24 07:16:49 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:49.887863 24105 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:49.888175 24105 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:16:49.888252 24105 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb485e706632411da22a0c8e60d79c7f, State: Initialized, Role: FOLLOWER
I20260924 07:16:49.888875 24105 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:49.889082 24105 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:49.889159 24105 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:49.889287 24105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:16:49.890275 24105 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:49.890650 24105 leader_election.cc:302] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb485e706632411da22a0c8e60d79c7f; no voters:
I20260924 07:16:49.891034 24105 leader_election.cc:288] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:16:49.891139 24110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:16:49.891319 24110 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 LEADER]: Becoming Leader. State: Replica: cb485e706632411da22a0c8e60d79c7f, State: Running, Role: LEADER
I20260924 07:16:49.891629 24110 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:49.892457 24105 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:49.892948 24110 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb485e706632411da22a0c8e60d79c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } } }
I20260924 07:16:49.893182 24110 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:49.893455 24110 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb485e706632411da22a0c8e60d79c7f. Latest consensus state: current_term: 1 leader_uuid: "cb485e706632411da22a0c8e60d79c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } } }
I20260924 07:16:49.893622 24110 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:49.893522 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:49.878269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.15.127.190:45483)
0924 07:16:49.878742 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:49.878759 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:49.878798 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:16:49.879612 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:49.879633 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:49.880013 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:49.880383 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:49.881259 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:49.882289 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:49.883468 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:49.883977 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:49.884098 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:49.887645 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:49.887651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:49.887669 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:49.887714 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:49.889870 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:49.890376 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:49.890382 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:49.890389 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:49.890578 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:49.890908 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:49.890912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:49.890914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:49.891486 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:49.891694 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:49.891976 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 07:16:49.892432 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":161,"threads_started":1}
W20260924 07:16:49.897850 24125 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:16:49.897929 24125 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:16:49.898006 24115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:49.898696 24115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:49.904471 24115 catalog_manager.cc:1347] Generated new cluster ID: 814a8ba253804ca69e726271e7d1a930
I20260924 07:16:49.904551 24115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:49.914495 24115 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:16:49.915735 24115 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:49.922457 24115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Generated new TSK 0
I20260924 07:16:49.923355 24115 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:16:49.958448 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45483
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:50.083978 24131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:50.084331 24131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:50.084434 24131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:50.088737 24131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:50.088858 24131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:50.088963 24131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:50.095052 24131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:50.096287 24131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:50.097455 24131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:50.107795 24137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.107795 24136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.107818 24139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:50.108489 24131 server_base.cc:1034] running on GCE node
I20260924 07:16:50.108882 24131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:50.109608 24131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:50.110800 24131 hybrid_clock.cc:648] HybridClock initialized: now 1790234210110772 us; error 41 us; skew 500 ppm
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:50.113788 24131 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:16:50.114694 24131 webserver.cc:460] Webserver started at http://127.15.127.129:43793/ using document root <none> and password file <none>
I20260924 07:16:50.115295 24131 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:50.115406 24131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:50.115634 24131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:50.117798 24131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "9660b669791d41e6ba9b1dce00e8fb7c"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "5dd597b5050af23abef1735660d85cec"
server_key_iv: "a6c50aa727ba9fdadc1a2724e17db4d2"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.118368 24131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "9660b669791d41e6ba9b1dce00e8fb7c"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "5dd597b5050af23abef1735660d85cec"
server_key_iv: "a6c50aa727ba9fdadc1a2724e17db4d2"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.122160 24131 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 07:16:50.124665 24146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.125977 24131 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:50.126112 24131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "9660b669791d41e6ba9b1dce00e8fb7c"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "5dd597b5050af23abef1735660d85cec"
server_key_iv: "a6c50aa727ba9fdadc1a2724e17db4d2"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.126235 24131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:50.158475 24131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:50.161787 24131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:50.161995 24131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:50.162611 24131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:50.163555 24131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:50.163627 24131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.163697 24131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:50.163738 24131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.172863 24131 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:35129
I20260924 07:16:50.172883 24259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:35129 every 8 connection(s)
I20260924 07:16:50.173839 24131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:50.176548 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24131
I20260924 07:16:50.176666 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:16:50.176985 15870 external_mini_cluster.cc:1428] Setting key 77ffbd9f2f20d81094db597c4af276c6
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:16:50.188206 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:50.175621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:34961 (local address 127.15.127.190:45483)
0924 07:16:50.175765 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:50.175768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:50.176607 (+ 839us) server_negotiation.cc:407] Connection header received
0924 07:16:50.178112 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:50.178115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:50.178169 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:50.178273 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:50.179513 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.180040 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.181280 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.181516 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.181629 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:50.184461 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:50.184464 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:50.184466 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:50.184492 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:50.186117 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:50.186630 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:50.186634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:50.186636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:50.186725 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:50.187073 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:50.187076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:50.187078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:50.187270 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:50.187378 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:50.187894 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 07:16:50.188038 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:16:50.189132 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:50.175987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:34961)
0924 07:16:50.176436 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:50.176485 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:50.177883 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:50.178451 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:50.178458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:50.178827 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:50.179361 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:50.179375 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.180170 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.180173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:50.181153 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:50.181162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.181671 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.181676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:50.181868 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:50.182487 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:50.182504 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:50.184280 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:50.186241 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:50.186247 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:50.186264 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:50.186525 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:50.186818 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:50.186823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:50.186825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:50.186972 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:50.187377 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:50.187385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:50.187652 (+ 267us) client_negotiation.cc:769] Sending connection context
0924 07:16:50.187855 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 07:16:50.188201 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":116,"threads_started":1}
I20260924 07:16:50.190610 24260 heartbeater.cc:347] Connected to a master server at 127.15.127.190:45483
I20260924 07:16:50.190892 24260 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:50.191438 24260 heartbeater.cc:511] Master 127.15.127.190:45483 requested a full tablet report, sending...
I20260924 07:16:50.193178 24069 ts_manager.cc:194] Registered new tserver with Master: 9660b669791d41e6ba9b1dce00e8fb7c (127.15.127.129:35129)
I20260924 07:16:50.194428 24069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:34961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:16:50.205824 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45483
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:50.320750 24265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:50.321120 24265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:50.321198 24265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:50.325362 24265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:50.325423 24265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:50.325484 24265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:50.330896 24265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:50.332230 24265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:50.333389 24265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:50.343482 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.343472 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:50.343564 24265 server_base.cc:1034] running on GCE node
W20260924 07:16:50.343743 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:50.344199 24265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:50.344877 24265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:50.346069 24265 hybrid_clock.cc:648] HybridClock initialized: now 1790234210346015 us; error 36 us; skew 500 ppm
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:50.348898 24265 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:16:50.349820 24265 webserver.cc:460] Webserver started at http://127.15.127.130:42219/ using document root <none> and password file <none>
I20260924 07:16:50.350389 24265 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:50.350449 24265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:50.350687 24265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:50.352773 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "6fa87f950eef4d5faae02eb3b82dfefc"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "b15652b714ca1fb06e6be3d2f17a47dc"
server_key_iv: "1c55692c348706b7346f7a3f1b0091d8"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.353431 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "6fa87f950eef4d5faae02eb3b82dfefc"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "b15652b714ca1fb06e6be3d2f17a47dc"
server_key_iv: "1c55692c348706b7346f7a3f1b0091d8"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.358667 24265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:16:50.361730 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.363108 24265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:50.363212 24265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "6fa87f950eef4d5faae02eb3b82dfefc"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "b15652b714ca1fb06e6be3d2f17a47dc"
server_key_iv: "1c55692c348706b7346f7a3f1b0091d8"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.363371 24265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:50.375147 24265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:50.379158 24265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:50.379383 24265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:50.380028 24265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:50.380885 24265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:50.380931 24265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.381078 24265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:50.381137 24265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.389693 24265 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:34975
I20260924 07:16:50.389711 24393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:34975 every 8 connection(s)
I20260924 07:16:50.390599 24265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:50.392513 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24265
I20260924 07:16:50.392632 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:16:50.392940 15870 external_mini_cluster.cc:1428] Setting key 9b7c789d3ee0359a4441c9f8db506df6
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:16:50.405435 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:50.392341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:59339 (local address 127.15.127.190:45483)
0924 07:16:50.392798 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:50.392801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:50.393702 (+ 901us) server_negotiation.cc:407] Connection header received
0924 07:16:50.395172 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:50.395178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:50.395237 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:50.395293 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:50.396626 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.397249 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.398422 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.398651 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.398764 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:50.401597 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:50.401600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:50.401602 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:50.401625 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:50.403157 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:50.403696 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:50.403700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:50.403701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:50.403802 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:50.404156 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:50.404160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:50.404162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:50.404351 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:50.404457 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:50.405148 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 07:16:50.405270 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362}
I20260924 07:16:50.406350 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:50.392872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:59339)
0924 07:16:50.393372 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:50.393568 (+ 196us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:50.394922 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:50.395640 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:50.395647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:50.396004 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:50.396474 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:50.396487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.397385 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.397390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:50.398294 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:50.398303 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.398803 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.398807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:50.398982 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:50.399606 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:50.399623 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:50.401407 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:50.403297 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:50.403303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:50.403320 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:50.403589 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:50.403896 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:50.403901 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:50.403903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:50.404057 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:50.404466 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:50.404475 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:50.404841 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 07:16:50.405097 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 07:16:50.405451 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":163,"threads_started":1}
I20260924 07:16:50.407793 24394 heartbeater.cc:347] Connected to a master server at 127.15.127.190:45483
I20260924 07:16:50.408097 24394 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:50.408612 24394 heartbeater.cc:511] Master 127.15.127.190:45483 requested a full tablet report, sending...
I20260924 07:16:50.409875 24069 ts_manager.cc:194] Registered new tserver with Master: 6fa87f950eef4d5faae02eb3b82dfefc (127.15.127.130:34975)
I20260924 07:16:50.410508 24069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:59339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:16:50.420532 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45483
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:50.535012 24399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:50.535288 24399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:50.535331 24399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:50.539510 24399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:50.539568 24399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:50.539628 24399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:50.545518 24399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:50.546578 24399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:50.547571 24399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:50.557068 24407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.557071 24404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.557071 24405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:50.557621 24399 server_base.cc:1034] running on GCE node
I20260924 07:16:50.558012 24399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:50.558656 24399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:50.559811 24399 hybrid_clock.cc:648] HybridClock initialized: now 1790234210559783 us; error 46 us; skew 500 ppm
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:50.562536 24399 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:16:50.563445 24399 webserver.cc:460] Webserver started at http://127.15.127.131:42777/ using document root <none> and password file <none>
I20260924 07:16:50.564055 24399 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:50.564154 24399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:50.564419 24399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:16:50.566486 24399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "9393cfbf6f7d4feabaa58d0bffe42299"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "2cb9fb7ae534a846b8b437c36784d64d"
server_key_iv: "85304232f8b293b70653091c4058c780"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.567103 24399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "9393cfbf6f7d4feabaa58d0bffe42299"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "2cb9fb7ae534a846b8b437c36784d64d"
server_key_iv: "85304232f8b293b70653091c4058c780"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.570860 24399 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:16:50.573262 24414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.574465 24399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:50.574584 24399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "9393cfbf6f7d4feabaa58d0bffe42299"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "2cb9fb7ae534a846b8b437c36784d64d"
server_key_iv: "85304232f8b293b70653091c4058c780"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.574692 24399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:50.614876 24399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:50.618424 24399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:50.618635 24399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:50.619289 24399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:50.620162 24399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:50.620226 24399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.620299 24399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:50.620339 24399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.629611 24399 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:33083
I20260924 07:16:50.629635 24527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:33083 every 8 connection(s)
I20260924 07:16:50.630832 24399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:50.638105 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24399
I20260924 07:16:50.638268 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:16:50.638559 15870 external_mini_cluster.cc:1428] Setting key 0693d150cf1e826c929e1de94daefc67
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:16:50.645949 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:50.633406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:54259 (local address 127.15.127.190:45483)
0924 07:16:50.633552 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:50.633556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:50.634631 (+ 1075us) server_negotiation.cc:407] Connection header received
0924 07:16:50.635451 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:50.635454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:50.635500 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:50.635575 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:50.636803 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.637428 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.638816 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.639147 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.639271 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:50.642373 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:50.642377 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:50.642381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:50.642407 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:50.643858 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:50.644391 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:50.644395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:50.644397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:50.644476 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:50.644812 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:50.644817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:50.644818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:50.645044 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:50.645142 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:50.645654 (+ 512us) server_negotiation.cc:299] Negotiation successful
0924 07:16:50.645780 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:16:50.646742 24530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:50.633806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:54259)
0924 07:16:50.634475 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:50.634516 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:50.635269 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:50.635715 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:50.635722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:50.636096 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:50.636671 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:50.636687 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.637539 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.637542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:50.638690 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:50.638706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:50.639314 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:50.639318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:50.639543 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:50.640421 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:50.640440 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:50.642204 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:16:50.643989 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:50.643996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:50.644011 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:50.644280 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:50.644568 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:16:50.644572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:16:50.644574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:16:50.644716 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:16:50.645153 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:16:50.645169 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:16:50.645416 (+ 247us) client_negotiation.cc:769] Sending connection context
0924 07:16:50.645601 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 07:16:50.645922 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":120,"threads_started":1}
I20260924 07:16:50.647802 24528 heartbeater.cc:347] Connected to a master server at 127.15.127.190:45483
I20260924 07:16:50.648011 24528 heartbeater.cc:464] Registering TS with master...
I20260924 07:16:50.648468 24528 heartbeater.cc:511] Master 127.15.127.190:45483 requested a full tablet report, sending...
I20260924 07:16:50.649477 24069 ts_manager.cc:194] Registered new tserver with Master: 9393cfbf6f7d4feabaa58d0bffe42299 (127.15.127.131:33083)
I20260924 07:16:50.650002 24069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:54259
I20260924 07:16:50.652366 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab.
Entry for principal oryx/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab.
WARNING: no policy specified for oryx/127.15.127.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab.
Entry for principal oryx/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab.
WARNING: no policy specified for oryx/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab.
Entry for principal oryx/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab.
I20260924 07:16:50.731289 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24131
I20260924 07:16:50.737493 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24265
I20260924 07:16:50.742842 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24399
I20260924 07:16:50.748306 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24037
I20260924 07:16:50.755223 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:45483
--webserver_interface=127.15.127.190
--webserver_port=34777
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:45483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:16:50.868593 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:50.868899 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:50.868942 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:50.873198 24537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:16:50.873272 24537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:50.873291 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:50.873309 24537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:16:50.873324 24537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:16:50.878945 24537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:45483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:45483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=34777
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:50.880134 24537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:50.881273 24537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:50.890163 24545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.890308 24542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:50.890272 24543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:50.890265 24537 server_base.cc:1034] running on GCE node
I20260924 07:16:50.891098 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:50.892325 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:50.893527 24537 hybrid_clock.cc:648] HybridClock initialized: now 1790234210893516 us; error 33 us; skew 500 ppm
Sep 24 07:16:50 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234210, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:50.896482 24537 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:16:50.897522 24537 webserver.cc:460] Webserver started at http://127.15.127.190:34777/ using document root <none> and password file <none>
I20260924 07:16:50.898124 24537 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:50.898249 24537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:50.902268 24537 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:16:50.904508 24552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:50.905828 24537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:16:50.905967 24537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "cb485e706632411da22a0c8e60d79c7f"
format_stamp: "Formatted at 2026-09-24 07:16:49 on dist-test-slave-zx46"
server_key: "0b1e9f00b7a9ade1f78e1c6bc2929b68"
server_key_iv: "bbe57a278377ea232abd24df78b91118"
server_key_version: "encryptionkey@0"
I20260924 07:16:50.906528 24537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:50.919461 24537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:50.922534 24537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:50.922747 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:50.929672 24537 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:45483
I20260924 07:16:50.929757 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:45483 every 8 connection(s)
I20260924 07:16:50.930827 24537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:16:50.931650 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24537
I20260924 07:16:50.932085 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:35129
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:45483
--webserver_port=43793
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:16:50.935923 24605 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:16:50.937641 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Bootstrap starting.
I20260924 07:16:50.954524 24605 log.cc:824] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:16:50.961598 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:16:50.962029 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Bootstrap complete.
I20260924 07:16:50.966604 24605 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:50.967068 24605 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb485e706632411da22a0c8e60d79c7f, State: Initialized, Role: FOLLOWER
I20260924 07:16:50.967594 24605 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:50.967757 24605 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:16:50.967839 24605 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:16:50.967960 24605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:16:50.970093 24605 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:50.970441 24605 leader_election.cc:302] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb485e706632411da22a0c8e60d79c7f; no voters:
I20260924 07:16:50.970798 24605 leader_election.cc:288] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:16:50.970889 24609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:16:50.971076 24609 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [term 2 LEADER]: Becoming Leader. State: Replica: cb485e706632411da22a0c8e60d79c7f, State: Running, Role: LEADER
I20260924 07:16:50.971395 24609 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } }
I20260924 07:16:50.972080 24605 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:16:50.972740 24610 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cb485e706632411da22a0c8e60d79c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } } }
I20260924 07:16:50.972867 24610 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:50.973564 24611 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb485e706632411da22a0c8e60d79c7f. Latest consensus state: current_term: 2 leader_uuid: "cb485e706632411da22a0c8e60d79c7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb485e706632411da22a0c8e60d79c7f" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 45483 } } }
I20260924 07:16:50.973707 24611 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f [sys.catalog]: This master's current role is: LEADER
I20260924 07:16:50.973914 24615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:16:50.977281 24615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:16:50.978444 24615 catalog_manager.cc:1259] Loaded cluster ID: 814a8ba253804ca69e726271e7d1a930
I20260924 07:16:50.978519 24615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:16:50.980885 24615 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:16:50.981567 24615 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb485e706632411da22a0c8e60d79c7f: Loaded TSK: 0
I20260924 07:16:50.982791 24615 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:16:51.051746 24607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:51.052044 24607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:51.052093 24607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:51.056434 24607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:51.056511 24607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:51.056574 24607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:16:51.062321 24607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.129.keytab
--principal=oryx/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:35129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=43793
--tserver_master_addrs=127.15.127.190:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:51.063485 24607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:51.064604 24607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:51.076082 24632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:51.076090 24631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:51.076474 24607 server_base.cc:1034] running on GCE node
W20260924 07:16:51.076548 24634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:51.076987 24607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:51.077699 24607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:51.078897 24607 hybrid_clock.cc:648] HybridClock initialized: now 1790234211078861 us; error 58 us; skew 500 ppm
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234211, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:51.082134 24607 init.cc:377] Logged in from keytab as oryx/127.15.127.129@KRBTEST.COM (short username oryx)
I20260924 07:16:51.083166 24607 webserver.cc:460] Webserver started at http://127.15.127.129:43793/ using document root <none> and password file <none>
I20260924 07:16:51.083779 24607 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:51.083904 24607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:51.088589 24607 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:51.091113 24641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.092628 24607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:16:51.092818 24607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "9660b669791d41e6ba9b1dce00e8fb7c"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "5dd597b5050af23abef1735660d85cec"
server_key_iv: "a6c50aa727ba9fdadc1a2724e17db4d2"
server_key_version: "encryptionkey@0"
I20260924 07:16:51.093516 24607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:51.112277 24607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:51.115878 24607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:51.116098 24607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:51.116715 24607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:51.117666 24607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:51.117735 24607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.117812 24607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:51.117863 24607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.126847 24607 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:35129
I20260924 07:16:51.126870 24754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:35129 every 8 connection(s)
I20260924 07:16:51.127836 24607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:16:51.129125 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24607
I20260924 07:16:51.129598 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:34975
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:45483
--webserver_port=42219
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:16:51.143059 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.131190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:42411 (local address 127.15.127.190:45483)
0924 07:16:51.131703 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.131723 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.131769 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:16:51.132524 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.132542 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.132963 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.133371 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.134634 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.136392 (+ 1758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.137701 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.138565 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.138780 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.142363 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:42411: BlockingRecv error: recv got EOF from 127.15.127.129:42411 (error 108)
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":125,"threads_started":1}
I20260924 07:16:51.143105 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.130178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:42411)
0924 07:16:51.130669 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.130723 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.132092 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.133476 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.133483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.133842 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.134451 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.134467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.136567 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.136570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.137569 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.137579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.138736 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.138739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.138871 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.139522 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.139539 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.142130 (+ 2591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
W20260924 07:16:51.143554 24755 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:45483 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.127.190:45483: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:51.149672 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.143991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:54821)
0924 07:16:51.144114 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.144129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.144229 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.144465 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.144468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.144642 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.144764 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.144770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.145563 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.145566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.146472 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.146482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.146908 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.146911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.147025 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.147482 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.147495 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.149516 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 07:16:51.149902 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.144103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:54821 (local address 127.15.127.190:45483)
0924 07:16:51.144251 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.144257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.144272 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:51.144311 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.144314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.144362 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.144429 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.144872 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.145460 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.146590 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.146810 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.146882 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.149645 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:54821: BlockingRecv error: recv got EOF from 127.15.127.129:54821 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:51.155762 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.150242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:40939)
0924 07:16:51.150374 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.150389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.150464 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.150740 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.150743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.150911 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.151014 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.151020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.151720 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.151723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.152589 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.152599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.153071 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.153075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.153167 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.153629 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.153643 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.155598 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 07:16:51.155977 24755 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:16:51.155870 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.150349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:40939 (local address 127.15.127.190:45483)
0924 07:16:51.150491 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.150496 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.150529 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:16:51.150568 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.150584 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.150638 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.150706 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.151132 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.151618 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.152717 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.152937 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.153111 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.155731 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:40939: BlockingRecv error: recv got EOF from 127.15.127.129:40939 (error 108)
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":112,"threads_started":1}
W20260924 07:16:51.250530 24758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:51.250831 24758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:51.250876 24758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:51.255070 24758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:51.255149 24758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:51.255250 24758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:16:51.260963 24758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.130.keytab
--principal=oryx/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:34975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=42219
--tserver_master_addrs=127.15.127.190:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:51.262166 24758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:51.263280 24758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:51.273373 24765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:51.273495 24758 server_base.cc:1034] running on GCE node
W20260924 07:16:51.273366 24766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:51.273685 24768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:51.274173 24758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:51.274854 24758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:51.276034 24758 hybrid_clock.cc:648] HybridClock initialized: now 1790234211276001 us; error 47 us; skew 500 ppm
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234211, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:51.279284 24758 init.cc:377] Logged in from keytab as oryx/127.15.127.130@KRBTEST.COM (short username oryx)
I20260924 07:16:51.280261 24758 webserver.cc:460] Webserver started at http://127.15.127.130:42219/ using document root <none> and password file <none>
I20260924 07:16:51.280908 24758 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:51.281042 24758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:51.285578 24758 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 07:16:51.287943 24775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.289355 24758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:16:51.289510 24758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "6fa87f950eef4d5faae02eb3b82dfefc"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "b15652b714ca1fb06e6be3d2f17a47dc"
server_key_iv: "1c55692c348706b7346f7a3f1b0091d8"
server_key_version: "encryptionkey@0"
I20260924 07:16:51.290078 24758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:51.314590 24758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:51.318081 24758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:51.318306 24758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:51.318938 24758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:51.319808 24758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:51.319890 24758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.319960 24758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:51.320004 24758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.329417 24758 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:34975
I20260924 07:16:51.329445 24888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:34975 every 8 connection(s)
I20260924 07:16:51.330475 24758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:16:51.337728 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24758
I20260924 07:16:51.338186 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:33083
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:45483
--webserver_port=42777
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:16:51.344233 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.332450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:56457 (local address 127.15.127.190:45483)
0924 07:16:51.332634 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.332638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.333479 (+ 841us) server_negotiation.cc:407] Connection header received
0924 07:16:51.335138 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.335142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.335219 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.335348 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.336755 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.337480 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.339044 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.339298 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.339443 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.344088 (+ 4645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:56457: BlockingRecv error: recv got EOF from 127.15.127.130:56457 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 07:16:51.345281 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.332824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:56457)
0924 07:16:51.333300 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.333347 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.334877 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.335575 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.335584 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.336009 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.336571 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.336587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.337663 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.337666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.338884 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.338896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.339519 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.339525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.339743 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.340472 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.340500 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.343928 (+ 3428us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":126,"threads_started":1}
W20260924 07:16:51.345865 24889 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:45483 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.127.190:45483: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:51.351665 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.346338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:37423)
0924 07:16:51.346461 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.346477 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.346569 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.346827 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.346830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.347005 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.347101 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.347107 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.347901 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.347906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.348474 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.348482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.348933 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.348937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.349058 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.349576 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.349588 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.351512 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 07:16:51.352097 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.346454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:37423 (local address 127.15.127.190:45483)
0924 07:16:51.346571 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.346576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.346591 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:51.346667 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.346670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.346722 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.346809 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.347249 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.347789 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.348593 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.348819 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.348928 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.351646 (+ 2718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:37423: BlockingRecv error: recv got EOF from 127.15.127.130:37423 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:51.357911 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.352252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:49499)
0924 07:16:51.352444 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.352458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.352538 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.352785 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.352788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.352949 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.353082 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.353088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.353815 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.353818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.354709 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.354718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.355143 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.355150 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.355271 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.355776 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.355790 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.357770 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260924 07:16:51.358079 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.352417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:49499 (local address 127.15.127.190:45483)
0924 07:16:51.352529 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.352532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.352544 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:16:51.352630 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.352634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.352671 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.352746 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.353209 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.353717 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.354825 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.355030 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.355108 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.357941 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:49499: BlockingRecv error: recv got EOF from 127.15.127.130:49499 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:16:51.358465 24889 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:16:51.456322 24892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:16:51.456640 24892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:16:51.456739 24892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:16:51.460955 24892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:16:51.461117 24892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:16:51.461215 24892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:16:51.466858 24892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/krb5kdc/oryx_127.15.127.131.keytab
--principal=oryx/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:33083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=42777
--tserver_master_addrs=127.15.127.190:45483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:16:51.468036 24892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:16:51.469182 24892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:16:51.479441 24898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:51.479554 24900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:16:51.479458 24897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:16:51.479585 24892 server_base.cc:1034] running on GCE node
I20260924 07:16:51.480247 24892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:16:51.480911 24892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:16:51.482117 24892 hybrid_clock.cc:648] HybridClock initialized: now 1790234211482098 us; error 30 us; skew 500 ppm
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234211, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:16:51.485224 24892 init.cc:377] Logged in from keytab as oryx/127.15.127.131@KRBTEST.COM (short username oryx)
I20260924 07:16:51.486049 24892 webserver.cc:460] Webserver started at http://127.15.127.131:42777/ using document root <none> and password file <none>
I20260924 07:16:51.486577 24892 fs_manager.cc:359] Metadata directory not provided
I20260924 07:16:51.486703 24892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:16:51.490641 24892 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 07:16:51.492910 24907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.494342 24892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:16:51.494477 24892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "9393cfbf6f7d4feabaa58d0bffe42299"
format_stamp: "Formatted at 2026-09-24 07:16:50 on dist-test-slave-zx46"
server_key: "2cb9fb7ae534a846b8b437c36784d64d"
server_key_iv: "85304232f8b293b70653091c4058c780"
server_key_version: "encryptionkey@0"
I20260924 07:16:51.495048 24892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:16:51.525300 24892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:16:51.528832 24892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:16:51.529094 24892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:16:51.529739 24892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:16:51.530593 24892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:16:51.530642 24892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.530710 24892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:16:51.530750 24892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:16:51.540159 24892 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:33083
I20260924 07:16:51.540176 25020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:33083 every 8 connection(s)
I20260924 07:16:51.541246 24892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:16:51.545465 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 24892
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:16:51.553850 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.543143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:44557 (local address 127.15.127.190:45483)
0924 07:16:51.543336 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.543341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.544266 (+ 925us) server_negotiation.cc:407] Connection header received
0924 07:16:51.546006 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.546012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.546087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.546186 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.547664 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.548281 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.549715 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.549999 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.550107 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.553735 (+ 3628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:44557: BlockingRecv error: recv got EOF from 127.15.127.131:44557 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260924 07:16:51.554683 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.543612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:44557)
0924 07:16:51.544090 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.544135 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.545576 (+ 1441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.546411 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.546421 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.546931 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.547504 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.547518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.548410 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.548413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.549596 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.549607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.550148 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.550150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.550303 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.550880 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.550896 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.553586 (+ 2690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
W20260924 07:16:51.555224 25021 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:45483 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.127.190:45483: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
I20260924 07:16:51.556067 24760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.546677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.15.127.190:45483)
0924 07:16:51.546828 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.546841 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.546854 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:51.546892 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.546897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.546940 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.547038 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.547865 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.548437 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.549425 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.549787 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.549865 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.551551 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:16:51.551556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:16:51.551573 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:16:51.551601 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:16:51.553582 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:51.554140 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:51.554145 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:51.554151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:51.554273 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:16:51.554565 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:16:51.554567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:16:51.554568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:16:51.555073 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:16:51.555273 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 07:16:51.555575 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 07:16:51.555963 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:51.561437 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.555691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:32991)
0924 07:16:51.555901 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.555917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.556000 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.556360 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.556363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.556520 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.556625 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.556632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.557533 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.557535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.558258 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.558267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.558862 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.558865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.558939 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.559454 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.559465 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.561291 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138}
W20260924 07:16:51.561515 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.555778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:32991 (local address 127.15.127.190:45483)
0924 07:16:51.555915 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.555920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.555959 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:16:51.556103 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.556106 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.556168 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.556250 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.556735 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.557409 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.558365 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.558716 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.558826 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.561403 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:32991: BlockingRecv error: recv got EOF from 127.15.127.131:32991 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:51 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:51.567042 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:51.562074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:43343)
0924 07:16:51.562208 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:51.562220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:51.562313 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:51.562549 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:51.562552 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:51.562715 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:51.562785 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.562790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.563495 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.563500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.564030 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:51.564037 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.564476 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.564480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:51.564557 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.565089 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:51.565103 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:51.566884 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 07:16:51.567124 24759 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:51.562195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:43343 (local address 127.15.127.190:45483)
0924 07:16:51.562356 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:51.562362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:51.562376 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:51.562413 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:51.562421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:51.562455 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:51.562535 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:51.562896 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.563390 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.564152 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:51.564369 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:51.564475 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:51.567004 (+ 2529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:43343: BlockingRecv error: recv got EOF from 127.15.127.131:43343 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260924 07:16:51.567318 25021 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 07:16:52 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:52 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:52.162253 25025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:52.156602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:39041)
0924 07:16:52.156869 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:52.156882 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:52.156969 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:52.157323 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:52.157326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:52.157517 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:52.157635 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:52.157641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.158598 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.158601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:52.159173 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:52.159181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.159683 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.159686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:52.159757 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:52.160278 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:52.160290 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:52.162136 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":83,"threads_started":1}
W20260924 07:16:52.162443 25026 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:52.156669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:39041 (local address 127.15.127.190:45483)
0924 07:16:52.156929 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:52.156932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:52.156945 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:52.157121 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:52.157123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:52.157171 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:52.157251 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:52.157855 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.158476 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.159298 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.159574 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.159670 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:52.162297 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:39041: BlockingRecv error: recv got EOF from 127.15.127.129:39041 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":93,"threads_started":1}
Sep 24 07:16:52 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:52 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:52.364548 25027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:52.359083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:59133)
0924 07:16:52.359374 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:52.359394 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:52.359523 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:52.359804 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:52.359807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:52.360024 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:52.360116 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:52.360121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.360895 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.360900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:52.361571 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:52.361580 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.362039 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.362043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:52.362127 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:52.362636 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:52.362647 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:52.364423 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":95,"threads_started":1}
W20260924 07:16:52.364688 25026 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:52.359152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:59133 (local address 127.15.127.190:45483)
0924 07:16:52.359326 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:52.359330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:52.359468 (+ 138us) server_negotiation.cc:407] Connection header received
0924 07:16:52.359629 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:52.359633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:52.359691 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:52.359779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:52.360243 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.360781 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.361699 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.361927 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.362038 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:52.364566 (+ 2528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:59133: BlockingRecv error: recv got EOF from 127.15.127.130:59133 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 07:16:52 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:52 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:52.573776 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:52.567923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:34909)
0924 07:16:52.568259 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:52.568272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:52.568412 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:52.568715 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:52.568719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:52.568922 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:52.569078 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:52.569086 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.569850 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.569855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:52.570437 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:52.570445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.570923 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.570927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:52.571013 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:52.571648 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:52.571660 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:52.573596 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":88,"threads_started":1}
W20260924 07:16:52.573834 25026 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:52.568000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:34909 (local address 127.15.127.190:45483)
0924 07:16:52.568188 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:52.568192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:52.568349 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:16:52.568516 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:52.568521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:52.568573 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:52.568691 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:52.569195 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.569738 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.570558 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:52.570808 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:52.570920 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:52.573723 (+ 2803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:34909: BlockingRecv error: recv got EOF from 127.15.127.131:34909 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 07:16:53 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:53 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:53.168256 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:53.163090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:55457)
0924 07:16:53.163370 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:53.163383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:53.163473 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:53.163728 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:53.163730 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:53.163905 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:53.164006 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:53.164011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.164743 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.164746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:53.165378 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:53.165388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.165837 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.165840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:53.165905 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:53.166388 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:53.166399 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:53.168113 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":89,"threads_started":1}
W20260924 07:16:53.168470 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:53.163194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:55457 (local address 127.15.127.190:45483)
0924 07:16:53.163421 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:53.163425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:53.163438 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:53.163563 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:53.163566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:53.163609 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:53.163690 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:53.164104 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.164636 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.165500 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.165732 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.165814 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:53.168329 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:55457: BlockingRecv error: recv got EOF from 127.15.127.129:55457 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":58,"threads_started":1}
Sep 24 07:16:53 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:53 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:53.370818 25031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:53.365396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:52005)
0924 07:16:53.365682 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:53.365696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:53.365783 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:53.366048 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:53.366050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:53.366228 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:53.366325 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:53.366330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.367061 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.367063 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:53.367599 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:53.367605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.368037 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.368040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:53.368126 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:53.368792 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:53.368803 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:53.370612 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":103,"threads_started":1}
W20260924 07:16:53.370838 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:53.365584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:52005 (local address 127.15.127.190:45483)
0924 07:16:53.365739 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:53.365743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:53.365758 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:53.365857 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:53.365859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:53.365905 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:53.366036 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:53.366454 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.366969 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.367698 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.367902 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.368031 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:53.370739 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:52005: BlockingRecv error: recv got EOF from 127.15.127.130:52005 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 07:16:53 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:53 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:53.580478 25032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:53.574593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:43323)
0924 07:16:53.574935 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:53.574952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:53.575111 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:53.575392 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:53.575395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:53.575609 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:53.575696 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:53.575701 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.576467 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.576475 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:53.577123 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:53.577132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.577611 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.577615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:53.577699 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:53.578270 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:53.578282 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:53.580281 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
W20260924 07:16:53.580535 25030 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:53.574773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:43323 (local address 127.15.127.190:45483)
0924 07:16:53.574936 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:53.574941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:53.575015 (+ 74us) server_negotiation.cc:407] Connection header received
0924 07:16:53.575233 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:53.575235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:53.575280 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:53.575364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:53.575809 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.576336 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.577249 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:53.577468 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:53.577599 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:53.580419 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:43323: BlockingRecv error: recv got EOF from 127.15.127.131:43323 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 07:16:54 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:54 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:54.175631 25033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:54.169161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:36069)
0924 07:16:54.169448 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:54.169461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:54.169563 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:54.169993 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:54.169996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:54.170190 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:54.170282 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:54.170287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.171101 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.171105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:54.171957 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:54.171967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.172794 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.172800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:54.172921 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:54.173512 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:54.173523 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:54.175443 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":111,"threads_started":1}
W20260924 07:16:54.175694 25034 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:54.169216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:36069 (local address 127.15.127.190:45483)
0924 07:16:54.169450 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:54.169453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:54.169529 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:16:54.169639 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:54.169642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:54.169714 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:54.169818 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:54.170409 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.170985 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.172081 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.172525 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.173174 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:54.175559 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:36069: BlockingRecv error: recv got EOF from 127.15.127.129:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
Sep 24 07:16:54 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:54 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:54.377146 25035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:54.371601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:57113)
0924 07:16:54.371875 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:54.371888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:54.371986 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:54.372285 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:54.372288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:54.372495 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:54.372582 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:54.372588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.373483 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.373489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:54.374082 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:54.374091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.374543 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.374546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:54.374632 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:54.375129 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:54.375141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:54.376964 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":105,"threads_started":1}
W20260924 07:16:54.377310 25034 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:54.371647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:57113 (local address 127.15.127.190:45483)
0924 07:16:54.371794 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:54.371798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:54.371953 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:16:54.372051 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:54.372053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:54.372095 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:54.372264 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:54.372705 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.373378 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.374206 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.374430 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.374540 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:54.377184 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:57113: BlockingRecv error: recv got EOF from 127.15.127.130:57113 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:16:54 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:54 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:16:54.587664 25034 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:54.581655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:60235 (local address 127.15.127.190:45483)
0924 07:16:54.581787 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:54.581792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:54.582008 (+ 216us) server_negotiation.cc:407] Connection header received
0924 07:16:54.582206 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:54.582210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:54.582286 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:54.582371 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:54.583025 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.583559 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.584392 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.584624 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.584740 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:54.587569 (+ 2829us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:60235: BlockingRecv error: recv got EOF from 127.15.127.131:60235 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:16:54.587675 25036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:54.581592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:60235)
0924 07:16:54.581897 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:54.581914 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:54.582058 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:54.582428 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:54.582434 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:54.582719 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:54.582876 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:54.582885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.583700 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.583704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:54.584269 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:54.584276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:54.584745 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:54.584749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:54.584833 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:54.585413 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:54.585433 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:54.587448 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":106,"threads_started":1}
Sep 24 07:16:55 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:55 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:55.187832 25037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:55.176540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:49439)
0924 07:16:55.176821 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:55.176835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:55.176930 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:55.177239 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:55.177242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:55.177422 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:55.177537 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:55.177543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.178345 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.178348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:55.178914 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:55.178922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.179453 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.179457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:55.179562 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:55.185468 (+ 5906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:55.185493 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:55.187643 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":93,"threads_started":1}
W20260924 07:16:55.187952 25038 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:55.176604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:49439 (local address 127.15.127.190:45483)
0924 07:16:55.176859 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:55.176863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:55.176879 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:16:55.177044 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:55.177046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:55.177091 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:55.177171 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:55.177650 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.178229 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.179048 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.179327 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.179419 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:55.187811 (+ 8392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:49439: BlockingRecv error: recv got EOF from 127.15.127.129:49439 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":75,"threads_started":1}
Sep 24 07:16:55 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:55 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:55.383322 25039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:55.378043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:55489)
0924 07:16:55.378305 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:55.378317 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:55.378413 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:55.378681 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:55.378684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:55.378860 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:55.378955 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:55.378960 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.379671 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.379674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:55.380248 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:55.380256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.380667 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.380669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:55.380732 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:55.381316 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:55.381327 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:55.383192 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
W20260924 07:16:55.383455 25038 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:55.378108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:55489 (local address 127.15.127.190:45483)
0924 07:16:55.378265 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:55.378268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:55.378382 (+ 114us) server_negotiation.cc:407] Connection header received
0924 07:16:55.378485 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:55.378488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:55.378531 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:55.378603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:55.379058 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.379567 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.380353 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.380561 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.380653 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:55.383336 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:55489: BlockingRecv error: recv got EOF from 127.15.127.130:55489 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 24 07:16:55 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:55 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:55.593760 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:55.588393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:53103)
0924 07:16:55.588683 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:55.588695 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:55.588825 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:55.589116 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:55.589118 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:55.589327 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:55.589417 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:55.589424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.590185 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.590191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:55.590753 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:55.590762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.591219 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.591223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:55.591309 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:55.591824 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:55.591835 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:55.593645 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
W20260924 07:16:55.593946 25038 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:55.588451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:53103 (local address 127.15.127.190:45483)
0924 07:16:55.588610 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:55.588614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:55.588771 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:16:55.588920 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:55.588924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:55.588977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:55.589093 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:55.589537 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.590062 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.590871 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:55.591099 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:55.591218 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:55.593799 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:53103: BlockingRecv error: recv got EOF from 127.15.127.131:53103 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 07:16:56 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:56 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:56.194183 25041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:56.188658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:43745)
0924 07:16:56.188943 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:56.188957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:56.189096 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:56.189336 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:56.189338 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:56.189518 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:56.189608 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:56.189613 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.190435 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.190437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:56.190985 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:56.190992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.191424 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.191426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:56.191487 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:56.192039 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:56.192051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:56.194058 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
W20260924 07:16:56.194393 25042 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:56.188793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:43745 (local address 127.15.127.190:45483)
0924 07:16:56.189105 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:56.189109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:56.189124 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:56.189164 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:56.189166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:56.189212 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:56.189303 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:56.189720 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.190315 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.191101 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.191322 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.191408 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:56.194239 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:43745: BlockingRecv error: recv got EOF from 127.15.127.129:43745 (error 108)
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":102,"threads_started":1}
Sep 24 07:16:56 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:56 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:56.389632 25043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:56.384193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:59837)
0924 07:16:56.384476 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:56.384492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:56.384639 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:56.384949 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:56.384955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:56.385204 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:56.385300 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:56.385306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.386015 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.386017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:56.386598 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:56.386606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.387019 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.387022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:56.387096 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:56.387596 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:56.387607 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:56.389514 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":108,"threads_started":1}
W20260924 07:16:56.389835 25042 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:56.384304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:59837 (local address 127.15.127.190:45483)
0924 07:16:56.384453 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:56.384457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:56.384582 (+ 125us) server_negotiation.cc:407] Connection header received
0924 07:16:56.384765 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:56.384768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:56.384820 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:56.384928 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:56.385403 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.385926 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.386710 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.386924 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.386997 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:56.389672 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:59837: BlockingRecv error: recv got EOF from 127.15.127.130:59837 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 07:16:56 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:56 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:56.600343 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:56.594699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:34565)
0924 07:16:56.594962 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:56.594975 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:56.595067 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:56.595276 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:56.595278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:56.595452 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:56.595540 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:56.595545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.596279 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.596284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:56.596866 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:56.596875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.597423 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.597427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:56.597500 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:56.597994 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:56.598005 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:56.600167 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
W20260924 07:16:56.600385 25042 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:56.594772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:34565 (local address 127.15.127.190:45483)
0924 07:16:56.594924 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:56.594929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:56.595031 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:16:56.595146 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:56.595149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:56.595195 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:56.595256 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:56.595640 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.596162 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.596980 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:56.597299 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:56.597384 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:56.600287 (+ 2903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:34565: BlockingRecv error: recv got EOF from 127.15.127.131:34565 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 07:16:57 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:57 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:57.200711 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:57.194902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:52777)
0924 07:16:57.195185 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:57.195198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:57.195290 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:57.195537 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:57.195539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:57.195721 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:57.195814 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:57.195818 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.196659 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.196665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:57.197353 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:57.197363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.197818 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.197821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:57.197880 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:57.198391 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:57.198412 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:57.200545 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
W20260924 07:16:57.200845 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:57.194966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:52777 (local address 127.15.127.190:45483)
0924 07:16:57.195222 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:57.195226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:57.195241 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:16:57.195380 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:57.195382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:57.195427 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:57.195504 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:57.195912 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.196519 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.197453 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.197694 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.197792 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:57.200698 (+ 2906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:52777: BlockingRecv error: recv got EOF from 127.15.127.129:52777 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":75,"threads_started":1}
Sep 24 07:16:57 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:57 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:57.395881 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:57.390429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:38939)
0924 07:16:57.390711 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:57.390725 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:57.390828 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:57.391052 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:57.391054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:57.391263 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:57.391355 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:57.391361 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.392201 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.392206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:57.392802 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:57.392809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.393313 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.393316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:57.393401 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:57.393940 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:57.393951 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:57.395761 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":107,"threads_started":1}
W20260924 07:16:57.396015 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:57.390489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:38939 (local address 127.15.127.190:45483)
0924 07:16:57.390639 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:57.390644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:57.390784 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:16:57.390907 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:57.390910 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:57.390953 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:57.391014 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:57.391459 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.392023 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.392925 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.393188 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.393296 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:57.395897 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:38939: BlockingRecv error: recv got EOF from 127.15.127.130:38939 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 24 07:16:57 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:57 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:57.606498 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:57.601132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:48237)
0924 07:16:57.601402 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:57.601415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:57.601552 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:57.601851 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:57.601854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:57.602055 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:57.602146 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:57.602152 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.602892 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.602898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:57.603518 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:57.603526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.603972 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.603976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:57.604062 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:57.604571 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:57.604583 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:57.606385 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
W20260924 07:16:57.606663 25046 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:57.601143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:48237 (local address 127.15.127.190:45483)
0924 07:16:57.601288 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:57.601293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:57.601492 (+ 199us) server_negotiation.cc:407] Connection header received
0924 07:16:57.601657 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:57.601661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:57.601718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:57.601803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:57.602263 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.602785 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.603635 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:57.603862 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:57.603970 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:57.606517 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:48237: BlockingRecv error: recv got EOF from 127.15.127.131:48237 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 07:16:58 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:58 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:58.206739 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:58.201500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:47279)
0924 07:16:58.201799 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:58.201812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:58.201897 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:58.202203 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:58.202205 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:58.202380 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:58.202475 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:58.202479 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.203235 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.203237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:58.203780 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:58.203787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.204255 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.204257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:58.204319 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:58.204784 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:58.204795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:58.206624 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
W20260924 07:16:58.206892 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:58.201589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:47279 (local address 127.15.127.190:45483)
0924 07:16:58.201842 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:58.201845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:58.201859 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:16:58.202016 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:58.202019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:58.202068 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:58.202149 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:58.202575 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.203117 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.203874 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.204091 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.204252 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:58.206756 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:47279: BlockingRecv error: recv got EOF from 127.15.127.129:47279 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":97,"threads_started":1}
Sep 24 07:16:58 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:58 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:58.403242 25051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:58.396779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:37045)
0924 07:16:58.397105 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:58.397119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:58.397221 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:58.397508 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:58.397510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:58.397702 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:58.397799 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:58.397804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.398871 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.398876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:58.399609 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:58.399619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.400209 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.400213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:58.400307 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:58.400813 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:58.400825 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:58.403076 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":79,"threads_started":1}
W20260924 07:16:58.403347 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:58.396870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:37045 (local address 127.15.127.190:45483)
0924 07:16:58.397081 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:58.397086 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:58.397193 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:16:58.397315 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:58.397318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:58.397365 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:58.397442 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:58.397903 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.398634 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.399728 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.400080 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.400170 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:58.403233 (+ 3063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:37045: BlockingRecv error: recv got EOF from 127.15.127.130:37045 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 07:16:58 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:58 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:58.613340 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:58.607261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:45441)
0924 07:16:58.607554 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:58.607567 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:58.607684 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:58.607913 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:58.607916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:58.608099 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:58.608195 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:58.608201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.608918 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.608920 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:58.609587 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:58.609596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.610137 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.610140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:58.610266 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:58.610935 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:58.610956 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:58.613225 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":67,"threads_started":1}
W20260924 07:16:58.613476 25050 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:58.607361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:45441 (local address 127.15.127.190:45483)
0924 07:16:58.607551 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:58.607555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:58.607623 (+ 68us) server_negotiation.cc:407] Connection header received
0924 07:16:58.607767 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:58.607769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:58.607816 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:58.607877 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:58.608305 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.608829 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.609693 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:58.610031 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:58.610143 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:58.613356 (+ 3213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:45441: BlockingRecv error: recv got EOF from 127.15.127.131:45441 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 07:16:59 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:59 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:59.213717 25053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:59.207601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:39459)
0924 07:16:59.207871 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:59.207884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:59.207972 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:59.208215 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:59.208217 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:59.208395 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:59.208487 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:59.208492 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.209298 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.209301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:59.210272 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:59.210282 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.210963 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.210967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:59.211086 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:59.211683 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:59.211697 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:59.213528 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
W20260924 07:16:59.213778 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:59.207783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:39459 (local address 127.15.127.190:45483)
0924 07:16:59.208010 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:59.208013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:59.208026 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:16:59.208060 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:59.208062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:59.208102 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:59.208182 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:59.208588 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.209198 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.210378 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.210721 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.210843 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:59.213658 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:39459: BlockingRecv error: recv got EOF from 127.15.127.129:39459 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
Sep 24 07:16:59 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:59 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:59.409762 25055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:59.404056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:50741)
0924 07:16:59.404315 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:59.404327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:59.404423 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:59.404647 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:59.404650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:59.404839 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:59.404935 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:59.404940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.405867 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.405869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:59.406409 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:59.406415 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.406825 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.406827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:59.406886 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:59.407461 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:59.407473 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:59.409589 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
W20260924 07:16:59.409861 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:59.404101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:50741 (local address 127.15.127.190:45483)
0924 07:16:59.404256 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:59.404259 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:59.404393 (+ 134us) server_negotiation.cc:407] Connection header received
0924 07:16:59.404497 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:59.404499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:59.404543 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:59.404612 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:59.405082 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.405768 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.406512 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.406729 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.406804 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:59.409737 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:50741: BlockingRecv error: recv got EOF from 127.15.127.130:50741 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 07:16:59 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:16:59 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:16:59.619295 25056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:16:59.614112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:44425)
0924 07:16:59.614424 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:16:59.614437 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:16:59.614537 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:16:59.614774 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:16:59.614776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:16:59.614951 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:16:59.615042 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:59.615046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.615757 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.615758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:59.616301 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:16:59.616308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.616717 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.616719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:16:59.616778 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:59.617277 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:16:59.617288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:16:59.619128 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":80,"threads_started":1}
W20260924 07:16:59.619362 25054 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:16:59.614217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:44425 (local address 127.15.127.190:45483)
0924 07:16:59.614353 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:16:59.614357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:16:59.614508 (+ 151us) server_negotiation.cc:407] Connection header received
0924 07:16:59.614619 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:16:59.614621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:16:59.614667 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:16:59.614736 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:16:59.615158 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.615667 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.616400 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:16:59.616621 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:16:59.616699 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:16:59.619254 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:44425: BlockingRecv error: recv got EOF from 127.15.127.131:44425 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 07:17:00 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:00 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:00.226197 25057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:00.214606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:48901)
0924 07:17:00.214878 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:00.214893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:00.214995 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:00.215272 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:00.215274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:00.215449 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:00.215590 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:00.215601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.216556 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.216559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:00.217241 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:00.217252 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.217664 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.217667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:00.217727 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:00.223997 (+ 6270us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:00.224012 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:00.226046 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":96,"threads_started":1}
W20260924 07:17:00.226418 25058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:00.214680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:48901 (local address 127.15.127.190:45483)
0924 07:17:00.214929 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:00.214932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:00.214952 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:17:00.215082 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:00.215084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:00.215128 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:00.215224 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:00.215712 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.216434 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.217343 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.217564 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.217643 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:00.226256 (+ 8613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:48901: BlockingRecv error: recv got EOF from 127.15.127.129:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":97,"threads_started":1}
Sep 24 07:17:00 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:00 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:00.416353 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:00.410611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:57057)
0924 07:17:00.410883 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:00.410896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:00.410993 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:00.411235 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:00.411237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:00.411410 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:00.411502 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:00.411507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.412439 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.412442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:00.413074 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:00.413085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.413596 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.413600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:00.413688 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:00.414177 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:00.414189 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:00.416157 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
W20260924 07:17:00.416384 25058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:00.410724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:57057 (local address 127.15.127.190:45483)
0924 07:17:00.410896 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:00.410900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:00.410959 (+ 59us) server_negotiation.cc:407] Connection header received
0924 07:17:00.411072 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:00.411074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:00.411118 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:00.411196 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:00.411638 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.412277 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.413203 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.413458 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.413580 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:00.416283 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:57057: BlockingRecv error: recv got EOF from 127.15.127.130:57057 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 07:17:00 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:00 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:00.626207 25060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:00.620308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:55103)
0924 07:17:00.620697 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:00.620710 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:00.620855 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:00.621218 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:00.621221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:00.621455 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:00.621561 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:00.621567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.622355 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.622358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:00.622911 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:00.622919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.623366 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.623369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:00.623433 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:00.624014 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:00.624034 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:00.626071 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
W20260924 07:17:00.626488 25058 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:00.620362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:55103 (local address 127.15.127.190:45483)
0924 07:17:00.620506 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:00.620510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:00.620796 (+ 286us) server_negotiation.cc:407] Connection header received
0924 07:17:00.620974 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:00.620996 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:00.621079 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:00.621174 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:00.621668 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.622217 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.623014 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:00.623246 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:00.623322 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:00.626238 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:55103: BlockingRecv error: recv got EOF from 127.15.127.131:55103 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 07:17:01 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:01 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:01.232635 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:01.227053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:40629)
0924 07:17:01.227366 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:01.227380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:01.227469 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:01.227714 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:01.227716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:01.227892 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:01.227986 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:01.227991 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.228689 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.228691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:01.229310 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:01.229319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.229726 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.229729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:01.229786 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:01.230341 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:01.230353 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:01.232467 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":131,"threads_started":1}
W20260924 07:17:01.232733 25062 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:01.227126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:40629 (local address 127.15.127.190:45483)
0924 07:17:01.227388 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:01.227392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:01.227434 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:17:01.227550 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:01.227552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:01.227595 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:01.227686 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:01.228087 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.228602 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.229412 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.229626 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.229711 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:01.232581 (+ 2870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:40629: BlockingRecv error: recv got EOF from 127.15.127.129:40629 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
Sep 24 07:17:01 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:01 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:01.422816 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:01.417376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:48133)
0924 07:17:01.417724 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:01.417738 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:01.417835 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:01.418037 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:01.418039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:01.418230 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:01.418327 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:01.418333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.419033 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.419035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:01.419659 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:01.419668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.420080 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.420082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:01.420159 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:01.420639 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:01.420653 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:01.422690 (+ 2037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":96,"threads_started":1}
W20260924 07:17:01.422981 25062 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:01.417433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:48133 (local address 127.15.127.190:45483)
0924 07:17:01.417567 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:01.417571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:01.417813 (+ 242us) server_negotiation.cc:407] Connection header received
0924 07:17:01.417918 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:01.417921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:01.417963 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:01.418020 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:01.418427 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.418937 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.419766 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.419981 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.420054 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:01.422851 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:48133: BlockingRecv error: recv got EOF from 127.15.127.130:48133 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 07:17:01 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:01 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:01.632189 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:01.626970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:46035)
0924 07:17:01.627274 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:01.627287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:01.627366 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:01.627587 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:01.627589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:01.627766 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:01.627857 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:01.627862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.628580 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.628582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:01.629209 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:01.629218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.629706 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.629709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:01.629778 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:01.630301 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:01.630313 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:01.632042 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
W20260924 07:17:01.632267 25062 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:01.627222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:46035 (local address 127.15.127.190:45483)
0924 07:17:01.627360 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:01.627364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:01.627385 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:17:01.627439 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:01.627442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:01.627486 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:01.627558 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:01.627961 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.628479 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.629326 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:01.629583 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:01.629666 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:01.632175 (+ 2509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:46035: BlockingRecv error: recv got EOF from 127.15.127.131:46035 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 07:17:02 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:02 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:02.239473 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:02.233605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:33671)
0924 07:17:02.233921 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:02.233935 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:02.234032 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:02.234356 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:02.234359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:02.234584 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:02.234730 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:02.234737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.235744 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.235747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:02.236308 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:02.236316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.236736 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.236739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:02.236801 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:02.237337 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:02.237348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:02.239295 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":94,"threads_started":1}
W20260924 07:17:02.239547 25066 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:02.233711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:33671 (local address 127.15.127.190:45483)
0924 07:17:02.234005 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:02.234008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:02.234023 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:17:02.234143 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:02.234147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:02.234198 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:02.234300 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:02.234858 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.235629 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.236409 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.236636 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.236715 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:02.239421 (+ 2706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:33671: BlockingRecv error: recv got EOF from 127.15.127.129:33671 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
Sep 24 07:17:02 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:02 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:02.429421 25067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:02.423743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:43217)
0924 07:17:02.423997 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:02.424010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:02.424168 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:02.424437 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:02.424440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:02.424646 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:02.424736 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:02.424741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.425559 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.425564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:02.426116 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:02.426123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.426598 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.426602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:02.426682 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:02.427273 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:02.427285 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:02.429248 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
W20260924 07:17:02.429483 25066 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:02.423775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:43217 (local address 127.15.127.190:45483)
0924 07:17:02.423931 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:02.423935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:02.424096 (+ 161us) server_negotiation.cc:407] Connection header received
0924 07:17:02.424273 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:02.424277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:02.424330 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:02.424418 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:02.424854 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.425453 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.426246 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.426486 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.426593 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:02.429377 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:43217: BlockingRecv error: recv got EOF from 127.15.127.130:43217 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 07:17:02 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:02 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:02.638943 25068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:02.633239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:41057)
0924 07:17:02.633513 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:02.633526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:02.633635 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:02.633951 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:02.633954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:02.634190 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:02.634321 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:02.634327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.635105 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.635112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:02.635700 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:02.635708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.636192 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.636195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:02.636284 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:02.636881 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:02.636893 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:02.638828 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
W20260924 07:17:02.639197 25066 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:02.633368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:41057 (local address 127.15.127.190:45483)
0924 07:17:02.633546 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:02.633551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:02.633589 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:17:02.633738 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:02.633741 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:02.633804 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:02.633924 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:02.634441 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.634981 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.635826 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:02.636076 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:02.636189 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:02.638977 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:41057: BlockingRecv error: recv got EOF from 127.15.127.131:41057 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 07:17:03 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:03 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:03.245959 25069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:03.240396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:57071)
0924 07:17:03.240700 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:03.240714 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:03.240817 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:03.241075 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:03.241077 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:03.241258 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:03.241352 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:03.241358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.242088 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.242091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:03.242648 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:03.242655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.243063 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.243065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:03.243125 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:03.243696 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:03.243707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:03.245784 (+ 2077us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":93,"threads_started":1}
W20260924 07:17:03.246037 25070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:03.240461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:57071 (local address 127.15.127.190:45483)
0924 07:17:03.240728 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:03.240731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:03.240787 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:17:03.240899 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:03.240902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:03.240948 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:03.241058 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:03.241461 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.241993 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.242748 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.242966 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.243042 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:03.245907 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:57071: BlockingRecv error: recv got EOF from 127.15.127.129:57071 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":61,"threads_started":1}
Sep 24 07:17:03 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:03 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:03.436232 25071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:03.430375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:58987)
0924 07:17:03.430668 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:03.430682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:03.430804 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:03.431050 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:03.431052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:03.431240 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:03.431339 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:03.431345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.432086 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.432089 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:03.432680 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:03.432688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.433258 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.433260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:03.433324 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:03.433815 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:03.433832 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:03.436081 (+ 2249us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":70,"threads_started":1}
W20260924 07:17:03.436378 25070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:03.430453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:58987 (local address 127.15.127.190:45483)
0924 07:17:03.430656 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:03.430661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:03.430766 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:17:03.430890 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:03.430893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:03.430936 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:03.431006 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:03.431444 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.431968 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.432828 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.433126 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.433244 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:03.436244 (+ 3000us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:58987: BlockingRecv error: recv got EOF from 127.15.127.130:58987 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 24 07:17:03 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:03 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:17:03.645532 25070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:03.640043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:37219 (local address 127.15.127.190:45483)
0924 07:17:03.640204 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:03.640211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:03.640317 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:17:03.640422 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:03.640424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:03.640470 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:03.640551 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:03.641027 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.641573 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.642354 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.642588 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.642695 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:03.645400 (+ 2705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:37219: BlockingRecv error: recv got EOF from 127.15.127.131:37219 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 07:17:03.645534 25072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:03.639936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:37219)
0924 07:17:03.640235 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:03.640247 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:03.640342 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:03.640576 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:03.640578 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:03.640770 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:03.640867 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:03.640872 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.641683 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.641685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:03.642245 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:03.642253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:03.642709 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:03.642711 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:03.642770 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:03.643316 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:03.643328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:03.645402 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":145,"threads_started":1}
Sep 24 07:17:04 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:04 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:04.252343 25073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:04.246976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:57781)
0924 07:17:04.247267 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:04.247281 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:04.247368 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:04.247712 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:04.247714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:04.247892 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:04.247989 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:04.247994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.248726 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.248728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:04.249368 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:04.249378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.249788 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.249791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:04.249855 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:04.250419 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:04.250432 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:04.252222 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
W20260924 07:17:04.252552 25074 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:04.247280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:57781 (local address 127.15.127.190:45483)
0924 07:17:04.247500 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:04.247504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:04.247515 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:17:04.247550 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:04.247552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:04.247593 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:04.247676 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:04.248090 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.248634 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.249472 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.249692 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.249768 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:04.252398 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:57781: BlockingRecv error: recv got EOF from 127.15.127.129:57781 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
Sep 24 07:17:04 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:04 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:04.442811 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:04.437159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:37345)
0924 07:17:04.437464 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:04.437479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:04.437584 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:04.437829 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:04.437831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:04.438038 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:04.438162 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:04.438168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.438903 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.438907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:04.439488 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:04.439497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.440036 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.440039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:04.440162 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:04.440661 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:04.440673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:04.442628 (+ 1955us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
W20260924 07:17:04.442907 25074 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:04.437242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:37345 (local address 127.15.127.190:45483)
0924 07:17:04.437390 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:04.437397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:04.437552 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:17:04.437672 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:04.437675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:04.437720 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:04.437789 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:04.438262 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.438791 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.439620 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.439894 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.440001 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:04.442795 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:37345: BlockingRecv error: recv got EOF from 127.15.127.130:37345 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 07:17:04 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:04 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:04.652432 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:04.646328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:44995)
0924 07:17:04.646657 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:04.646671 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:04.646813 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:04.647097 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:04.647100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:04.647326 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:04.647416 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:04.647421 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.648193 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.648198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:04.648784 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:04.648792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.649348 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.649352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:04.649440 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:04.649993 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:04.650004 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:04.652266 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":84,"threads_started":1}
W20260924 07:17:04.652521 25074 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:04.646447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:44995 (local address 127.15.127.190:45483)
0924 07:17:04.646618 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:04.646623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:04.646756 (+ 133us) server_negotiation.cc:407] Connection header received
0924 07:17:04.646921 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:04.646926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:04.646981 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:04.647078 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:04.647536 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.648062 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.648908 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:04.649220 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:04.649334 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:04.652426 (+ 3092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:44995: BlockingRecv error: recv got EOF from 127.15.127.131:44995 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 24 07:17:05 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:05 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:05.263274 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:05.253139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:46483)
0924 07:17:05.253450 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:05.253466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:05.253564 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:05.253797 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:05.253799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:05.253973 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:05.254072 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:05.254077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.254807 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.254810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:05.255363 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:05.255371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.255782 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.255784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:05.255845 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:05.260865 (+ 5020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:05.260879 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:05.263091 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
W20260924 07:17:05.263392 25078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:05.253214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:46483 (local address 127.15.127.190:45483)
0924 07:17:05.253557 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:05.253560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:05.253573 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:17:05.253632 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:05.253634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:05.253682 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:05.253759 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:05.254174 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.254710 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.255461 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.255689 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.255761 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:05.263241 (+ 7480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:46483: BlockingRecv error: recv got EOF from 127.15.127.129:46483 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":78,"threads_started":1}
Sep 24 07:17:05 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:05 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:17:05.450284 25078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:05.443690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:42871 (local address 127.15.127.190:45483)
0924 07:17:05.443872 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:05.443879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:05.443973 (+ 94us) server_negotiation.cc:407] Connection header received
0924 07:17:05.444073 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:05.444078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:05.444148 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:05.444260 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:05.444663 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.445698 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.446509 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.446958 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.447086 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:05.450192 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:42871: BlockingRecv error: recv got EOF from 127.15.127.130:42871 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:17:05.450292 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:05.443547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:42871)
0924 07:17:05.443881 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:05.443894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:05.443990 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:05.444276 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:05.444278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:05.444447 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:05.444539 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:05.444544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.445828 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.445831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:05.446386 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:05.446394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.447088 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.447095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:05.447193 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:05.447804 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:05.447825 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:05.450073 (+ 2248us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":83,"threads_started":1}
Sep 24 07:17:05 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:05 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.131@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:05.658664 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:05.653256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.131:48719)
0924 07:17:05.653519 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:05.653532 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:05.653629 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:05.653872 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:05.653874 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:05.654065 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:05.654194 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:05.654200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.654907 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.654910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:05.655493 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:05.655501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.655933 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.655936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:05.655999 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:05.656596 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:05.656608 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:05.658504 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
W20260924 07:17:05.658744 25078 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:05.653366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:48719 (local address 127.15.127.190:45483)
0924 07:17:05.653504 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:05.653507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:05.653583 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:17:05.653712 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:05.653714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:05.653759 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:05.653821 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:05.654297 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.654818 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.655595 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:05.655823 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:05.655901 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:05.658601 (+ 2700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.131:48719: BlockingRecv error: recv got EOF from 127.15.127.131:48719 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 07:17:06 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:06 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.129@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:06.276044 25081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:06.264345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.129:60479)
0924 07:17:06.264637 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:06.264651 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:06.264741 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:06.264975 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:06.264977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:06.265220 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:06.265319 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:06.265325 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.266073 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.266075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:06.266647 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:06.266655 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.267158 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.267160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:06.267223 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:06.273828 (+ 6605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:06.273844 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:06.275919 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":113,"threads_started":1}
W20260924 07:17:06.276275 25082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:06.264499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60479 (local address 127.15.127.190:45483)
0924 07:17:06.264735 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:06.264739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:06.264753 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:17:06.264814 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:06.264816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:06.264859 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:06.264937 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:06.265435 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.265972 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.266777 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.267012 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.267150 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:06.276089 (+ 8939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.129:60479: BlockingRecv error: recv got EOF from 127.15.127.129:60479 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":80,"threads_started":1}
Sep 24 07:17:06 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:17:06 dist-test-slave-zx46 krb5kdc[24024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.127.130@KRBTEST.COM for oryx/127.15.127.190@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:17:06.456987 25083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:06.451321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:45483 (local address 127.15.127.130:59969)
0924 07:17:06.451629 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:06.451651 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:06.451764 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:06.452065 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:06.452068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:06.452278 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:06.452366 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:06.452371 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.453216 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.453221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:06.453802 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:06.453810 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.454301 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.454304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:06.454389 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:06.454953 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:06.454967 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:06.456808 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.127.190:45483: Server oryx/127.15.127.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
W20260924 07:17:06.457077 25082 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:06.451699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:59969 (local address 127.15.127.190:45483)
0924 07:17:06.451853 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:06.451857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:06.451870 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:17:06.451905 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:06.451908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:06.451957 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:06.452043 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:06.452486 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.453093 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.453923 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.454180 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.454288 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:06.456936 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.127.130:59969: BlockingRecv error: recv got EOF from 127.15.127.130:59969 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:17:06.553947 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24607
I20260924 07:17:06.561878 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24758
I20260924 07:17:06.567278 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24892
I20260924 07:17:06.572925 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 24537
2026-09-24T07:17:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18091 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T07:17:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:17:06Z Disabled control of system clock
I20260924 07:17:06.603406 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:38443
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:38443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:17:06.718286 25087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:06.718596 25087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:06.718637 25087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:06.723218 25087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:17:06.723301 25087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:06.723320 25087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:06.723340 25087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:17:06.723357 25087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:17:06.728570 25087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:38443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:38443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:06.729712 25087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:06.731043 25087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:06.739943 25093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:06.739969 25087 server_base.cc:1034] running on GCE node
W20260924 07:17:06.740091 25092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:06.740211 25095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:06.740793 25087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:06.742007 25087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:06.743283 25087 hybrid_clock.cc:648] HybridClock initialized: now 1790234226743251 us; error 44 us; skew 500 ppm
I20260924 07:17:06.745415 25087 webserver.cc:460] Webserver started at http://127.15.127.190:44181/ using document root <none> and password file <none>
I20260924 07:17:06.746027 25087 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:06.746094 25087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:06.746326 25087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:06.748435 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d"
format_stamp: "Formatted at 2026-09-24 07:17:06 on dist-test-slave-zx46"
server_key: "27b3522376df491dedc01be8cccd79e6"
server_key_iv: "296624bb5bebd83d239238cfb412c99c"
server_key_version: "encryptionkey@0"
I20260924 07:17:06.749104 25087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d"
format_stamp: "Formatted at 2026-09-24 07:17:06 on dist-test-slave-zx46"
server_key: "27b3522376df491dedc01be8cccd79e6"
server_key_iv: "296624bb5bebd83d239238cfb412c99c"
server_key_version: "encryptionkey@0"
I20260924 07:17:06.752789 25087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:17:06.755321 25101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:06.756521 25087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:17:06.756661 25087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d"
format_stamp: "Formatted at 2026-09-24 07:17:06 on dist-test-slave-zx46"
server_key: "27b3522376df491dedc01be8cccd79e6"
server_key_iv: "296624bb5bebd83d239238cfb412c99c"
server_key_version: "encryptionkey@0"
I20260924 07:17:06.756785 25087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:06.778417 25087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:06.779179 25087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:06.779397 25087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:06.786681 25087 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:38443
I20260924 07:17:06.786688 25153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:38443 every 8 connection(s)
I20260924 07:17:06.787683 25087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:17:06.789554 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25087
I20260924 07:17:06.789671 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:17:06.789990 15870 external_mini_cluster.cc:1428] Setting key 0d9978095cf56337c7ea31c2e6e753cc
I20260924 07:17:06.791538 25154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:06.796909 25154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d: Bootstrap starting.
I20260924 07:17:06.799145 25154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:06.799353 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:06.791414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.15.127.190:38443)
0924 07:17:06.792071 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:06.792086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:06.792122 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:17:06.792793 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:06.792810 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:06.793295 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:06.793653 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:06.794119 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.795122 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.796015 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:06.796487 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:06.796677 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:06.797636 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:06.797642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:06.797655 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:06.798033 (+ 378us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:06.798086 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:06.798106 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:06.798218 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:06.798341 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 07:17:06.798598 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
I20260924 07:17:06.799947 25154 log.cc:824] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:06.802044 25154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d: No bootstrap required, opened a new log
I20260924 07:17:06.805531 25154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38443 } }
I20260924 07:17:06.805779 25154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:06.805877 25154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c8db1cdc9a949a5a594f7dae4bcce9d, State: Initialized, Role: FOLLOWER
I20260924 07:17:06.806329 25154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38443 } }
I20260924 07:17:06.806454 25154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:17:06.806542 25154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:17:06.806644 25154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:06.807555 25154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38443 } }
I20260924 07:17:06.807893 25154 leader_election.cc:302] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c8db1cdc9a949a5a594f7dae4bcce9d; no voters:
I20260924 07:17:06.808207 25154 leader_election.cc:288] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:17:06.808332 25159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:17:06.808591 25159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [term 1 LEADER]: Becoming Leader. State: Replica: 8c8db1cdc9a949a5a594f7dae4bcce9d, State: Running, Role: LEADER
I20260924 07:17:06.808938 25159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38443 } }
I20260924 07:17:06.809489 25154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:17:06.810593 25160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38443 } } }
I20260924 07:17:06.810739 25160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:06.811167 25168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:17:06.811113 25161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c8db1cdc9a949a5a594f7dae4bcce9d. Latest consensus state: current_term: 1 leader_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c8db1cdc9a949a5a594f7dae4bcce9d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38443 } } }
I20260924 07:17:06.811579 25161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:06.814540 25168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:17:06.820627 25168 catalog_manager.cc:1347] Generated new cluster ID: f9a78d7a94bd4f81b497f315420d45fc
I20260924 07:17:06.820708 25168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:17:06.830070 25168 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:17:06.830989 25168 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:17:06.849612 25168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c8db1cdc9a949a5a594f7dae4bcce9d: Generated new TSK 0
I20260924 07:17:06.850417 25168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:17:06.861325 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38443
--builtin_ntp_servers=127.15.127.148:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 07:17:06.975602 25178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:06.975895 25178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:06.975932 25178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:06.980140 25178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:06.980199 25178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:06.980252 25178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:17:06.985167 25178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:06.986290 25178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:06.987859 25178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:06.997850 25186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:06.997843 25183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:06.997843 25184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:06.998229 25178 server_base.cc:1034] running on GCE node
I20260924 07:17:06.998653 25178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:06.999343 25178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:07.000540 25178 hybrid_clock.cc:648] HybridClock initialized: now 1790234227000508 us; error 57 us; skew 500 ppm
I20260924 07:17:07.002470 25178 webserver.cc:460] Webserver started at http://127.15.127.129:43759/ using document root <none> and password file <none>
I20260924 07:17:07.003067 25178 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:07.003166 25178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:07.003405 25178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:07.005566 25178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "2feb3a93baa041d698d74622b8f7ab15"
format_stamp: "Formatted at 2026-09-24 07:17:06 on dist-test-slave-zx46"
server_key: "1405deca64b89d95ce062cdbaa43f397"
server_key_iv: "bf4473b3b5448587db1306521db2e4fb"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.006160 25178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "2feb3a93baa041d698d74622b8f7ab15"
format_stamp: "Formatted at 2026-09-24 07:17:06 on dist-test-slave-zx46"
server_key: "1405deca64b89d95ce062cdbaa43f397"
server_key_iv: "bf4473b3b5448587db1306521db2e4fb"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.010008 25178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 07:17:07.012755 25192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.014077 25178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:17:07.014227 25178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "2feb3a93baa041d698d74622b8f7ab15"
format_stamp: "Formatted at 2026-09-24 07:17:06 on dist-test-slave-zx46"
server_key: "1405deca64b89d95ce062cdbaa43f397"
server_key_iv: "bf4473b3b5448587db1306521db2e4fb"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.014349 25178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:07.054596 25178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:07.055321 25178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:07.055524 25178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:07.056129 25178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:07.057133 25178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:07.057199 25178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.057267 25178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:07.057319 25178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.066111 25178 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:32869
I20260924 07:17:07.066131 25305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:32869 every 8 connection(s)
I20260924 07:17:07.067052 25178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:17:07.068277 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25178
I20260924 07:17:07.068398 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:17:07.068724 15870 external_mini_cluster.cc:1428] Setting key 3e2ff4e04e92b7bfe42c06f18069d9bd
I20260924 07:17:07.070919 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38443
--builtin_ntp_servers=127.15.127.148:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:17:07.077858 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.069328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:55769 (local address 127.15.127.190:38443)
0924 07:17:07.069473 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:07.069477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:07.070900 (+ 1423us) server_negotiation.cc:407] Connection header received
0924 07:17:07.072971 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:07.072975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:07.073055 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:07.073162 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:07.074081 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.074622 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.075787 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.076029 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.076142 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:07.077341 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:07.077345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:07.077349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:07.077367 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:07.077386 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:07.077394 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:07.077474 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:07.077639 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 07:17:07.077713 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:17:07.078742 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.069825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38443 (local address 127.15.127.129:55769)
0924 07:17:07.070700 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:07.070761 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:07.072733 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:07.073343 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:07.073351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:07.073439 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:07.073933 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:07.073945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.074749 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.074752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:07.075661 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:07.075670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.076185 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.076189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:07.076366 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:07.077042 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:17:07.077045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:07.077107 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:17:07.077176 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:07.077476 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:07.077489 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:07.077495 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:17:07.077640 (+ 145us) client_negotiation.cc:240] Negotiation successful
0924 07:17:07.077932 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":157,"threads_started":1}
I20260924 07:17:07.080183 25306 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38443
I20260924 07:17:07.080432 25306 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:07.080963 25306 heartbeater.cc:511] Master 127.15.127.190:38443 requested a full tablet report, sending...
I20260924 07:17:07.082669 25118 ts_manager.cc:194] Registered new tserver with Master: 2feb3a93baa041d698d74622b8f7ab15 (127.15.127.129:32869)
I20260924 07:17:07.084226 25118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.129:55769
W20260924 07:17:07.184183 25309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:07.184459 25309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:07.184496 25309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:07.188611 25309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:07.188685 25309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:07.188741 25309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:17:07.193634 25309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:07.194890 25309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:07.196256 25309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:07.206230 25314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.206216 25315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.206216 25317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:07.206509 25309 server_base.cc:1034] running on GCE node
I20260924 07:17:07.206957 25309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:07.207602 25309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:07.208771 25309 hybrid_clock.cc:648] HybridClock initialized: now 1790234227208750 us; error 30 us; skew 500 ppm
I20260924 07:17:07.210786 25309 webserver.cc:460] Webserver started at http://127.15.127.130:37135/ using document root <none> and password file <none>
I20260924 07:17:07.211416 25309 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:07.211484 25309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:07.211727 25309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:07.213866 25309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "ce2df5017edb49d9b9738a556dbc376c"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "64d34f89157fa4a9c1a235b02d090e41"
server_key_iv: "420331bbf0305afe461f4cded4bbd7cc"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.214424 25309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "ce2df5017edb49d9b9738a556dbc376c"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "64d34f89157fa4a9c1a235b02d090e41"
server_key_iv: "420331bbf0305afe461f4cded4bbd7cc"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.218156 25309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:17:07.220657 25323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.221912 25309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:17:07.222043 25309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "ce2df5017edb49d9b9738a556dbc376c"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "64d34f89157fa4a9c1a235b02d090e41"
server_key_iv: "420331bbf0305afe461f4cded4bbd7cc"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.222158 25309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:07.251893 25309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:07.252715 25309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:07.252945 25309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:07.253629 25309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:07.254642 25309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:07.254727 25309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.254843 25309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:07.254901 25309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.264518 25309 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:36473
I20260924 07:17:07.264533 25436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:36473 every 8 connection(s)
I20260924 07:17:07.265582 25309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:17:07.267846 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25309
I20260924 07:17:07.267983 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:17:07.268381 15870 external_mini_cluster.cc:1428] Setting key 4ef965a33f558e83eb881f9a0723246b
I20260924 07:17:07.270567 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38443
--builtin_ntp_servers=127.15.127.148:44723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:17:07.275892 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.267469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:45631 (local address 127.15.127.190:38443)
0924 07:17:07.267618 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:07.267621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:07.268970 (+ 1349us) server_negotiation.cc:407] Connection header received
0924 07:17:07.270512 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:07.270516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:07.270571 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:07.270681 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:07.271686 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.272391 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.273685 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.273949 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.274065 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:07.275418 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:07.275421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:07.275423 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:07.275436 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:07.275454 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:07.275462 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:07.275519 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:07.275722 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:17:07.275778 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 07:17:07.277287 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.267996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38443 (local address 127.15.127.130:45631)
0924 07:17:07.268753 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:07.268824 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:07.270295 (+ 1471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:07.270922 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:07.270929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:07.271022 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:07.271535 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:07.271548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.272528 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.272532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:07.273522 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:07.273532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.274147 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.274153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:07.274408 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:07.275078 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:17:07.275082 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:07.275155 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:17:07.275253 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:07.275551 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:07.275563 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:07.275573 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 07:17:07.275724 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 07:17:07.276082 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":145,"threads_started":1}
I20260924 07:17:07.278483 25437 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38443
I20260924 07:17:07.278760 25437 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:07.279469 25437 heartbeater.cc:511] Master 127.15.127.190:38443 requested a full tablet report, sending...
I20260924 07:17:07.280602 25118 ts_manager.cc:194] Registered new tserver with Master: ce2df5017edb49d9b9738a556dbc376c (127.15.127.130:36473)
I20260924 07:17:07.281265 25118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.130:45631
W20260924 07:17:07.384814 25440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:07.385144 25440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:07.385216 25440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:07.389195 25440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:07.389281 25440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:07.389385 25440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:17:07.394049 25440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:07.395284 25440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:07.396628 25440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:17:07.406122 25440 server_base.cc:1034] running on GCE node
W20260924 07:17:07.406103 25446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.406100 25445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.406376 25448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:07.406762 25440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:07.407383 25440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:07.408571 25440 hybrid_clock.cc:648] HybridClock initialized: now 1790234227408554 us; error 28 us; skew 500 ppm
I20260924 07:17:07.410437 25440 webserver.cc:460] Webserver started at http://127.15.127.131:40727/ using document root <none> and password file <none>
I20260924 07:17:07.411144 25440 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:07.411235 25440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:07.411443 25440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:07.413661 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "e58a042def824333b6d792a61b1f942a"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "6d189a1ad7684b99ed8da886a4331420"
server_key_iv: "2898dd3b3e027082187072026365f644"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.414225 25440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "e58a042def824333b6d792a61b1f942a"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "6d189a1ad7684b99ed8da886a4331420"
server_key_iv: "2898dd3b3e027082187072026365f644"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.418004 25440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:17:07.420535 25454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.421870 25440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:17:07.422012 25440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "e58a042def824333b6d792a61b1f942a"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "6d189a1ad7684b99ed8da886a4331420"
server_key_iv: "2898dd3b3e027082187072026365f644"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.422135 25440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:07.443315 25440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:07.444038 25440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:07.444254 25440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:07.444844 25440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:07.445868 25440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:07.445938 25440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.446009 25440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:07.446058 25440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.454499 25440 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:43735
I20260924 07:17:07.454515 25567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:43735 every 8 connection(s)
I20260924 07:17:07.455423 25440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:17:07.456815 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25440
I20260924 07:17:07.456930 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:17:07.457273 15870 external_mini_cluster.cc:1428] Setting key 4732b030fd4261b3c7a782ac8e193e0a
I20260924 07:17:07.464854 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.457292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:54119 (local address 127.15.127.190:38443)
0924 07:17:07.457461 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:07.457465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:07.458816 (+ 1351us) server_negotiation.cc:407] Connection header received
0924 07:17:07.460339 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:07.460342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:07.460386 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:07.460478 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:07.461432 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.461966 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.463060 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.463265 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.463341 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:07.464369 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:07.464372 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:07.464384 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:07.464408 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:07.464432 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:07.464452 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:07.464529 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:07.464668 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:17:07.464725 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 07:17:07.465790 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.457803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38443 (local address 127.15.127.131:54119)
0924 07:17:07.458654 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:07.458709 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:07.460154 (+ 1445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:07.460651 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:07.460658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:07.460743 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:07.461308 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:07.461322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.462070 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.462073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:07.462958 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:07.462967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:07.463389 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:07.463391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:07.463531 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:07.464088 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:17:07.464090 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:07.464137 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:17:07.464203 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:07.464538 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:07.464544 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:07.464548 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:17:07.464648 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:17:07.464879 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":153,"threads_started":1}
I20260924 07:17:07.466990 25568 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38443
I20260924 07:17:07.467262 25568 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:07.467808 25568 heartbeater.cc:511] Master 127.15.127.190:38443 requested a full tablet report, sending...
I20260924 07:17:07.468847 25118 ts_manager.cc:194] Registered new tserver with Master: e58a042def824333b6d792a61b1f942a (127.15.127.131:43735)
I20260924 07:17:07.469456 25118 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.131:54119
I20260924 07:17:07.471246 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:17:07.474139 25157 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:07.473280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.15.127.190:38443)
0924 07:17:07.473431 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:07.473435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:07.473610 (+ 175us) server_negotiation.cc:407] Connection header received
0924 07:17:07.473704 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:07.473706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:07.473744 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:07.473807 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:07.474015 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:17:07.474256 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25178
I20260924 07:17:07.480443 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25309
I20260924 07:17:07.485498 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25440
I20260924 07:17:07.490492 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25087
2026-09-24T07:17:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (916 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T07:17:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:17:07Z Disabled control of system clock
I20260924 07:17:07.520028 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41973
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:41973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:17:07.632164 25579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:07.632447 25579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:07.632483 25579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:07.636610 25579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:17:07.636679 25579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:07.636695 25579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:07.636713 25579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:17:07.636727 25579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:17:07.641587 25579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:41973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:07.642681 25579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:07.644042 25579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:07.652563 25585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.652580 25584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.652568 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:07.652796 25579 server_base.cc:1034] running on GCE node
I20260924 07:17:07.653515 25579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:07.654621 25579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:07.655823 25579 hybrid_clock.cc:648] HybridClock initialized: now 1790234227655800 us; error 36 us; skew 500 ppm
I20260924 07:17:07.657761 25579 webserver.cc:460] Webserver started at http://127.15.127.190:46207/ using document root <none> and password file <none>
I20260924 07:17:07.658360 25579 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:07.658454 25579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:07.658695 25579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:07.660776 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "115585e50a874f70ab1121e35ff180c0"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "f300b4f487f17356161eab511fbf5066"
server_key_iv: "17a6e4fb3cacb4db5442b0feed7c6846"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.661460 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "115585e50a874f70ab1121e35ff180c0"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "f300b4f487f17356161eab511fbf5066"
server_key_iv: "17a6e4fb3cacb4db5442b0feed7c6846"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.665226 25579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:17:07.667582 25593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.668788 25579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:17:07.668929 25579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "115585e50a874f70ab1121e35ff180c0"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "f300b4f487f17356161eab511fbf5066"
server_key_iv: "17a6e4fb3cacb4db5442b0feed7c6846"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.669112 25579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:07.673753 25579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:07.674360 25579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:07.674538 25579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:07.681227 25579 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:41973
I20260924 07:17:07.681248 25645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:41973 every 8 connection(s)
I20260924 07:17:07.682175 25579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:17:07.685585 25646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:07.685660 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25579
I20260924 07:17:07.685762 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:17:07.686077 15870 external_mini_cluster.cc:1428] Setting key d92a9edeaddb597c3c34817b35957a4c
I20260924 07:17:07.691938 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.687635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41362 (local address 127.15.127.190:41973)
0924 07:17:07.688270 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:07.688284 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:07.688319 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:17:07.689085 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:07.689110 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:07.689487 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:07.689856 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:07.690680 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:07.690684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:07.690696 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:07.690706 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:07.690762 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:07.690777 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:07.690932 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:07.691000 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:17:07.691242 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":133,"threads_started":1}
I20260924 07:17:07.692602 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0: Bootstrap starting.
I20260924 07:17:07.694962 25646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:07.695696 25646 log.cc:824] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:07.697943 25646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0: No bootstrap required, opened a new log
I20260924 07:17:07.702428 25646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115585e50a874f70ab1121e35ff180c0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41973 } }
I20260924 07:17:07.702705 25646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:07.702772 25646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 115585e50a874f70ab1121e35ff180c0, State: Initialized, Role: FOLLOWER
I20260924 07:17:07.703279 25646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115585e50a874f70ab1121e35ff180c0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41973 } }
I20260924 07:17:07.703480 25646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:17:07.703550 25646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:17:07.703647 25646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:07.704591 25646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115585e50a874f70ab1121e35ff180c0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41973 } }
I20260924 07:17:07.704944 25646 leader_election.cc:302] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 115585e50a874f70ab1121e35ff180c0; no voters:
I20260924 07:17:07.705302 25646 leader_election.cc:288] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:17:07.705504 25651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:17:07.705824 25651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [term 1 LEADER]: Becoming Leader. State: Replica: 115585e50a874f70ab1121e35ff180c0, State: Running, Role: LEADER
I20260924 07:17:07.706171 25651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115585e50a874f70ab1121e35ff180c0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41973 } }
I20260924 07:17:07.706600 25646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:17:07.708108 25653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 115585e50a874f70ab1121e35ff180c0. Latest consensus state: current_term: 1 leader_uuid: "115585e50a874f70ab1121e35ff180c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115585e50a874f70ab1121e35ff180c0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41973 } } }
I20260924 07:17:07.708235 25653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:07.708639 25662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:17:07.708591 25652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "115585e50a874f70ab1121e35ff180c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "115585e50a874f70ab1121e35ff180c0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41973 } } }
I20260924 07:17:07.708676 25652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:07.711447 25662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:17:07.716938 25662 catalog_manager.cc:1347] Generated new cluster ID: 4953768d97f1474e90e0e6f38ffa4038
I20260924 07:17:07.717046 25662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:17:07.728909 25662 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:17:07.729312 25662 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:17:07.737756 25662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 115585e50a874f70ab1121e35ff180c0: Generated new TSK 0
I20260924 07:17:07.738438 25662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:17:07.749552 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41973
--builtin_ntp_servers=127.15.127.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:17:07.859957 25670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:07.860263 25670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:07.860342 25670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:07.864382 25670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:07.864477 25670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:07.864570 25670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:17:07.869323 25670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:07.870476 25670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:07.871917 25670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:07.881896 25675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.881867 25676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:07.881868 25678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:07.882076 25670 server_base.cc:1034] running on GCE node
I20260924 07:17:07.882539 25670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:07.883216 25670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:07.884425 25670 hybrid_clock.cc:648] HybridClock initialized: now 1790234227884411 us; error 42 us; skew 500 ppm
I20260924 07:17:07.886320 25670 webserver.cc:460] Webserver started at http://127.15.127.129:40693/ using document root <none> and password file <none>
I20260924 07:17:07.886912 25670 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:07.887004 25670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:07.887240 25670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:07.889423 25670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "1c1320b321cd47b1bb43bd10ba5554c4"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "704c1e1b9a54984177164ad8a5f73713"
server_key_iv: "3d1e6b5a1a86afba1c10a0142c9e4944"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.890012 25670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "1c1320b321cd47b1bb43bd10ba5554c4"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "704c1e1b9a54984177164ad8a5f73713"
server_key_iv: "3d1e6b5a1a86afba1c10a0142c9e4944"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.893746 25670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:17:07.896273 25684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.897562 25670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:17:07.897706 25670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "1c1320b321cd47b1bb43bd10ba5554c4"
format_stamp: "Formatted at 2026-09-24 07:17:07 on dist-test-slave-zx46"
server_key: "704c1e1b9a54984177164ad8a5f73713"
server_key_iv: "3d1e6b5a1a86afba1c10a0142c9e4944"
server_key_version: "encryptionkey@0"
I20260924 07:17:07.897821 25670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:07.902416 25670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:07.903012 25670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:07.903204 25670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:07.903733 25670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:07.904603 25670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:07.904681 25670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.904749 25670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:07.904784 25670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:07.913671 25670 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:40191
I20260924 07:17:07.913691 25797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:40191 every 8 connection(s)
I20260924 07:17:07.914526 25670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:17:07.915695 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25670
I20260924 07:17:07.915802 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:17:07.916110 15870 external_mini_cluster.cc:1428] Setting key 5a663431b07eb26b5d3c60f28fdd1d39
I20260924 07:17:07.918305 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41973
--builtin_ntp_servers=127.15.127.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 07:17:07.921665 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.916464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:54509 (local address 127.15.127.190:41973)
0924 07:17:07.916797 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:07.916802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:07.917905 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 07:17:07.919801 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:07.919807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:07.919859 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:07.919949 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:07.921143 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:07.921148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:07.921154 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:07.921170 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:07.921195 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:07.921208 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:07.921305 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:07.921426 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:17:07.921472 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260924 07:17:07.922493 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:07.916917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41973 (local address 127.15.127.129:54509)
0924 07:17:07.917751 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:07.917806 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:07.919542 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:07.920124 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:07.920131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:07.920216 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:07.920841 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:17:07.920843 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:07.920882 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:17:07.920948 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:07.921293 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:07.921299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:07.921303 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:17:07.921433 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 07:17:07.921620 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
I20260924 07:17:07.923650 25798 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41973
I20260924 07:17:07.923863 25798 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:07.924260 25798 heartbeater.cc:511] Master 127.15.127.190:41973 requested a full tablet report, sending...
I20260924 07:17:07.925859 25610 ts_manager.cc:194] Registered new tserver with Master: 1c1320b321cd47b1bb43bd10ba5554c4 (127.15.127.129:40191)
W20260924 07:17:08.031092 25801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:08.031394 25801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:08.031476 25801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:08.035584 25801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:08.035665 25801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:08.035756 25801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:17:08.040509 25801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:08.041653 25801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:08.043051 25801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:17:08.053146 25801 server_base.cc:1034] running on GCE node
W20260924 07:17:08.053062 25806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:08.053066 25807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:08.053066 25809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:08.053805 25801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:08.054483 25801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:08.055671 25801 hybrid_clock.cc:648] HybridClock initialized: now 1790234228055656 us; error 27 us; skew 500 ppm
I20260924 07:17:08.057523 25801 webserver.cc:460] Webserver started at http://127.15.127.130:40943/ using document root <none> and password file <none>
I20260924 07:17:08.058111 25801 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:08.058236 25801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:08.058458 25801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:08.060479 25801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "056a7c3738e24c2ea914ecaec179ade9"
format_stamp: "Formatted at 2026-09-24 07:17:08 on dist-test-slave-zx46"
server_key: "56d55b06c311bffe7df5e4f97e16bc1f"
server_key_iv: "8de9987c422cea16323dd8ad9187cdd9"
server_key_version: "encryptionkey@0"
I20260924 07:17:08.061106 25801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "056a7c3738e24c2ea914ecaec179ade9"
format_stamp: "Formatted at 2026-09-24 07:17:08 on dist-test-slave-zx46"
server_key: "56d55b06c311bffe7df5e4f97e16bc1f"
server_key_iv: "8de9987c422cea16323dd8ad9187cdd9"
server_key_version: "encryptionkey@0"
I20260924 07:17:08.064708 25801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:17:08.067195 25815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:08.068377 25801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:17:08.068507 25801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "056a7c3738e24c2ea914ecaec179ade9"
format_stamp: "Formatted at 2026-09-24 07:17:08 on dist-test-slave-zx46"
server_key: "56d55b06c311bffe7df5e4f97e16bc1f"
server_key_iv: "8de9987c422cea16323dd8ad9187cdd9"
server_key_version: "encryptionkey@0"
I20260924 07:17:08.068619 25801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:08.073237 25801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:08.073936 25801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:08.074127 25801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:08.074661 25801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:08.075516 25801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:08.075588 25801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:08.075661 25801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:08.075713 25801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:08.084182 25801 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:39935
I20260924 07:17:08.084205 25928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:39935 every 8 connection(s)
I20260924 07:17:08.085032 25801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:17:08.091050 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:08.086769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:49225 (local address 127.15.127.190:41973)
0924 07:17:08.086905 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:08.086909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:08.087946 (+ 1037us) server_negotiation.cc:407] Connection header received
0924 07:17:08.089304 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:08.089308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:08.089355 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:08.089462 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:08.090575 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:08.090578 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:08.090581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:08.090593 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:08.090616 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:08.090631 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:08.090726 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:08.090872 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:17:08.090918 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:17:08.091850 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:08.087180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41973 (local address 127.15.127.130:49225)
0924 07:17:08.087803 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:08.087839 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:08.089100 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:08.089621 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:08.089634 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:08.089721 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:08.090319 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:17:08.090321 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:08.090358 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:17:08.090424 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:08.090720 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:08.090728 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:08.090737 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:17:08.090915 (+ 178us) client_negotiation.cc:240] Negotiation successful
0924 07:17:08.091147 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260924 07:17:08.092954 25929 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41973
I20260924 07:17:08.093204 25929 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:08.093585 25929 heartbeater.cc:511] Master 127.15.127.190:41973 requested a full tablet report, sending...
I20260924 07:17:08.094534 25610 ts_manager.cc:194] Registered new tserver with Master: 056a7c3738e24c2ea914ecaec179ade9 (127.15.127.130:39935)
I20260924 07:17:08.094915 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25801
I20260924 07:17:08.095045 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:17:08.095360 15870 external_mini_cluster.cc:1428] Setting key 7cff712ce93b95d457dfced3543c9635
I20260924 07:17:08.097177 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41973
--builtin_ntp_servers=127.15.127.148:43565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:17:08.208667 25932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:08.209043 25932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:08.209138 25932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:08.213270 25932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:08.213321 25932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:08.213378 25932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:17:08.218663 25932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:43565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:08.219691 25932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:08.221131 25932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:08.231036 25937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:08.231045 25938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:08.231045 25940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:08.231097 25932 server_base.cc:1034] running on GCE node
I20260924 07:17:08.231660 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:08.232296 25932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:08.233474 25932 hybrid_clock.cc:648] HybridClock initialized: now 1790234228233434 us; error 54 us; skew 500 ppm
I20260924 07:17:08.235323 25932 webserver.cc:460] Webserver started at http://127.15.127.131:42211/ using document root <none> and password file <none>
I20260924 07:17:08.235948 25932 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:08.236037 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:08.236253 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:08.238427 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "0113a1538ad44081b69c3fca069b212f"
format_stamp: "Formatted at 2026-09-24 07:17:08 on dist-test-slave-zx46"
server_key: "d2faf3da5aa8af7b2b1fde29a1ae62a4"
server_key_iv: "9f9edc3116753f3e3a9d6c6260d39f9b"
server_key_version: "encryptionkey@0"
I20260924 07:17:08.239007 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "0113a1538ad44081b69c3fca069b212f"
format_stamp: "Formatted at 2026-09-24 07:17:08 on dist-test-slave-zx46"
server_key: "d2faf3da5aa8af7b2b1fde29a1ae62a4"
server_key_iv: "9f9edc3116753f3e3a9d6c6260d39f9b"
server_key_version: "encryptionkey@0"
I20260924 07:17:08.242660 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:17:08.245321 25946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:08.246651 25932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:17:08.246824 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "0113a1538ad44081b69c3fca069b212f"
format_stamp: "Formatted at 2026-09-24 07:17:08 on dist-test-slave-zx46"
server_key: "d2faf3da5aa8af7b2b1fde29a1ae62a4"
server_key_iv: "9f9edc3116753f3e3a9d6c6260d39f9b"
server_key_version: "encryptionkey@0"
I20260924 07:17:08.246963 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:08.251863 25932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:08.252543 25932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:08.252764 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:08.253427 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:08.254416 25932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:08.254496 25932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:08.254581 25932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:08.254629 25932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:08.264626 25932 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:41969
I20260924 07:17:08.264657 26059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:41969 every 8 connection(s)
I20260924 07:17:08.265677 25932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:17:08.271931 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:08.267650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:57575 (local address 127.15.127.190:41973)
0924 07:17:08.267787 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:08.267791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:08.268826 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 07:17:08.270148 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:08.270151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:08.270192 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:08.270275 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:08.271409 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:08.271411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:08.271414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:17:08.271423 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:08.271438 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:17:08.271444 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:08.271575 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:08.271750 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 07:17:08.271797 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:17:08.272876 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:08.268005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41973 (local address 127.15.127.131:57575)
0924 07:17:08.268656 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:08.268696 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:08.269930 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:08.270408 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:08.270415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:08.270512 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:08.271130 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:17:08.271132 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:08.271173 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:17:08.271242 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:08.271567 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:08.271576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:08.271583 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:17:08.271784 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:17:08.272039 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
I20260924 07:17:08.273303 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 25932
I20260924 07:17:08.273404 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:17:08.273797 15870 external_mini_cluster.cc:1428] Setting key f8d0d9f0708285510135f4038b84488e
I20260924 07:17:08.274073 26060 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41973
I20260924 07:17:08.274335 26060 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:08.274822 26060 heartbeater.cc:511] Master 127.15.127.190:41973 requested a full tablet report, sending...
I20260924 07:17:08.275884 25610 ts_manager.cc:194] Registered new tserver with Master: 0113a1538ad44081b69c3fca069b212f (127.15.127.131:41969)
I20260924 07:17:08.287640 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:17:08.290585 25649 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:17:08.289634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.15.127.190:41973)
0924 07:17:08.289801 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:08.289807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:08.289948 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:17:08.290101 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:08.290105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:08.290163 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:08.290251 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:08.290484 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41368: BlockingRecv error: recv got EOF from 127.0.0.1:41368 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:17:08.291067 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25670
I20260924 07:17:08.296566 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25801
I20260924 07:17:08.303269 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25932
I20260924 07:17:08.308583 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 25579
2026-09-24T07:17:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (818 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:17:08 dist-test-slave-zx46 krb5kdc[26070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:17:08 dist-test-slave-zx46 krb5kdc[26070](info): set up 2 sockets
Sep 24 07:17:08 dist-test-slave-zx46 krb5kdc[26070](info): commencing operation
krb5kdc: starting...
W20260924 07:17:09.353986 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:17:09 dist-test-slave-zx46 krb5kdc[26070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:17:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:17:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:17:09.478207 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:38489
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:09.590185 26083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:09.590458 26083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:09.590534 26083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:09.594769 26083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:17:09.594826 26083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:09.594846 26083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:09.594887 26083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:17:09.594916 26083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:17:09.600291 26083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:38489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:09.601467 26083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:09.602509 26083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:09.610872 26088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:09.610872 26089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:09.610848 26091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:09.611163 26083 server_base.cc:1034] running on GCE node
I20260924 07:17:09.611799 26083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:09.612862 26083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:09.614017 26083 hybrid_clock.cc:648] HybridClock initialized: now 1790234229613996 us; error 34 us; skew 500 ppm
Sep 24 07:17:09 dist-test-slave-zx46 krb5kdc[26070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:09.616748 26083 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:17:09.617690 26083 webserver.cc:460] Webserver started at http://127.15.127.190:41591/ using document root <none> and password file <none>
I20260924 07:17:09.618255 26083 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:09.618346 26083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:09.618575 26083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:09.620643 26083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "542335cb9cae473cac080b2aeed4413c"
format_stamp: "Formatted at 2026-09-24 07:17:09 on dist-test-slave-zx46"
server_key: "ba8845074e190988da319f64e2afb5e2"
server_key_iv: "831f6a501af0abc93663defdc312569c"
server_key_version: "encryptionkey@0"
I20260924 07:17:09.621246 26083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "542335cb9cae473cac080b2aeed4413c"
format_stamp: "Formatted at 2026-09-24 07:17:09 on dist-test-slave-zx46"
server_key: "ba8845074e190988da319f64e2afb5e2"
server_key_iv: "831f6a501af0abc93663defdc312569c"
server_key_version: "encryptionkey@0"
I20260924 07:17:09.624945 26083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:17:09.627344 26098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:09.628578 26083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:17:09.628715 26083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "542335cb9cae473cac080b2aeed4413c"
format_stamp: "Formatted at 2026-09-24 07:17:09 on dist-test-slave-zx46"
server_key: "ba8845074e190988da319f64e2afb5e2"
server_key_iv: "831f6a501af0abc93663defdc312569c"
server_key_version: "encryptionkey@0"
I20260924 07:17:09.628829 26083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:09.644474 26083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:09.647850 26083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:09.648054 26083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:09.654888 26083 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:38489
I20260924 07:17:09.655195 26150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:38489 every 8 connection(s)
I20260924 07:17:09.655905 26083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:17:09.658749 26151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:09.664268 26151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c: Bootstrap starting.
I20260924 07:17:09.664319 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 26083
I20260924 07:17:09.664428 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:17:09.664753 15870 external_mini_cluster.cc:1428] Setting key 90a26f2d643323a2f01bb54ec8859fc8
I20260924 07:17:09.666955 26151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:09.667692 26151 log.cc:824] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:09.669567 26151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c: No bootstrap required, opened a new log
I20260924 07:17:09.672924 26151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542335cb9cae473cac080b2aeed4413c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38489 } }
I20260924 07:17:09.673223 26151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:09.673302 26151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542335cb9cae473cac080b2aeed4413c, State: Initialized, Role: FOLLOWER
Sep 24 07:17:09 dist-test-slave-zx46 krb5kdc[26070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:17:09.673797 26151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542335cb9cae473cac080b2aeed4413c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38489 } }
I20260924 07:17:09.673985 26151 raft_consensus.cc:388] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:17:09.674059 26151 raft_consensus.cc:482] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:17:09.674162 26151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:09.675043 26151 raft_consensus.cc:504] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542335cb9cae473cac080b2aeed4413c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38489 } }
I20260924 07:17:09.675387 26151 leader_election.cc:302] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 542335cb9cae473cac080b2aeed4413c; no voters:
I20260924 07:17:09.675734 26151 leader_election.cc:288] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:17:09.675807 26156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:17:09.675983 26156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [term 1 LEADER]: Becoming Leader. State: Replica: 542335cb9cae473cac080b2aeed4413c, State: Running, Role: LEADER
I20260924 07:17:09.676342 26156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542335cb9cae473cac080b2aeed4413c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38489 } }
I20260924 07:17:09.677057 26151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:17:09.678195 26157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 542335cb9cae473cac080b2aeed4413c. Latest consensus state: current_term: 1 leader_uuid: "542335cb9cae473cac080b2aeed4413c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542335cb9cae473cac080b2aeed4413c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38489 } } }
I20260924 07:17:09.678200 26158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "542335cb9cae473cac080b2aeed4413c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542335cb9cae473cac080b2aeed4413c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38489 } } }
I20260924 07:17:09.678323 26157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:09.678344 26158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:09.678819 26165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:17:09.680261 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:09.666151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.15.127.190:38489)
0924 07:17:09.666574 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:09.666589 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:09.666641 (+ 52us) server_negotiation.cc:407] Connection header received
0924 07:17:09.667366 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:09.667386 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:09.667731 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:09.668111 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:09.668933 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:09.670070 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:09.670940 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:09.671420 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:09.671596 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:09.674284 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:09.674292 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:09.674310 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:09.674341 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:09.676439 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:09.676906 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:09.676910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:09.676915 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:09.677079 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:09.677433 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:09.677436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:09.677438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:09.677971 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:09.678788 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:09.678955 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:17:09.679246 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":141,"threads_started":1}
I20260924 07:17:09.683079 26165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:17:09.688695 26165 catalog_manager.cc:1347] Generated new cluster ID: f17f2ee2dbd949318baf53e1e5227e45
I20260924 07:17:09.688763 26165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:17:09.704845 26165 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:17:09.706422 26165 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:17:09.712452 26165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c: Generated new TSK 0
I20260924 07:17:09.713250 26165 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:17:09.744150 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38489
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:09.855218 26177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:09.855537 26177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:09.855625 26177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:09.859751 26177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:09.859839 26177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:09.859931 26177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:17:09.865221 26177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:09.866290 26177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:09.867345 26177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:09.876940 26183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:09.876935 26182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:09.876940 26185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:09.876978 26177 server_base.cc:1034] running on GCE node
I20260924 07:17:09.877765 26177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:09.878489 26177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:09.879665 26177 hybrid_clock.cc:648] HybridClock initialized: now 1790234229879632 us; error 47 us; skew 500 ppm
Sep 24 07:17:09 dist-test-slave-zx46 krb5kdc[26070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:09.882496 26177 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:17:09.883339 26177 webserver.cc:460] Webserver started at http://127.15.127.129:40851/ using document root <none> and password file <none>
I20260924 07:17:09.883898 26177 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:09.883988 26177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:09.884238 26177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:09.886313 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "b9dc99bb5e754aea8f7b8fef88216c14"
format_stamp: "Formatted at 2026-09-24 07:17:09 on dist-test-slave-zx46"
server_key: "7d57807d3f7ab49e6786e5fa52dc47ed"
server_key_iv: "e70974014908e476f6c3a4f253826f62"
server_key_version: "encryptionkey@0"
I20260924 07:17:09.886878 26177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "b9dc99bb5e754aea8f7b8fef88216c14"
format_stamp: "Formatted at 2026-09-24 07:17:09 on dist-test-slave-zx46"
server_key: "7d57807d3f7ab49e6786e5fa52dc47ed"
server_key_iv: "e70974014908e476f6c3a4f253826f62"
server_key_version: "encryptionkey@0"
I20260924 07:17:09.890573 26177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:17:09.893038 26192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:09.894305 26177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:17:09.894447 26177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "b9dc99bb5e754aea8f7b8fef88216c14"
format_stamp: "Formatted at 2026-09-24 07:17:09 on dist-test-slave-zx46"
server_key: "7d57807d3f7ab49e6786e5fa52dc47ed"
server_key_iv: "e70974014908e476f6c3a4f253826f62"
server_key_version: "encryptionkey@0"
I20260924 07:17:09.894564 26177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:09.921177 26177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:09.924069 26177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:09.924291 26177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:09.924892 26177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:09.925732 26177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:09.925804 26177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:09.925874 26177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:09.925925 26177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:09.935412 26177 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:41913
I20260924 07:17:09.935437 26305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:41913 every 8 connection(s)
I20260924 07:17:09.936311 26177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:17:09.940943 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 26177
I20260924 07:17:09.941107 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:17:09.941390 15870 external_mini_cluster.cc:1428] Setting key 577daa5715509eb44daccfd078f66dc7
Sep 24 07:17:09 dist-test-slave-zx46 krb5kdc[26070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:17:09.951617 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:09.938136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60019 (local address 127.15.127.190:38489)
0924 07:17:09.938277 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:09.938281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:09.939083 (+ 802us) server_negotiation.cc:407] Connection header received
0924 07:17:09.940496 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:09.940500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:09.940562 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:09.940673 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:09.942040 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:09.942841 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:09.944618 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:09.945046 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:09.945199 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:09.947855 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:09.947858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:09.947860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:09.947882 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:09.949504 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:09.950057 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:09.950062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:09.950064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:09.950144 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:09.950482 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:09.950488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:09.950493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:09.950680 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:09.950785 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:09.951325 (+ 540us) server_negotiation.cc:299] Negotiation successful
0924 07:17:09.951450 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:17:09.952440 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:09.938497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38489 (local address 127.15.127.129:60019)
0924 07:17:09.938930 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:09.938973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:09.940274 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:09.940854 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:09.940861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:09.941336 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:09.941876 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:09.941898 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:09.942965 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:09.942968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:09.944478 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:09.944493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:09.945226 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:09.945229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:09.945373 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:09.945990 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:09.946015 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:09.947682 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:09.949631 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:09.949637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:09.949655 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:09.949921 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:09.950236 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:09.950241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:09.950243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:09.950380 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:09.950783 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:09.950790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:09.951045 (+ 255us) client_negotiation.cc:769] Sending connection context
0924 07:17:09.951272 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 07:17:09.951614 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:17:09.953789 26306 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38489
I20260924 07:17:09.954061 26306 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:09.954576 26306 heartbeater.cc:511] Master 127.15.127.190:38489 requested a full tablet report, sending...
I20260924 07:17:09.956255 26115 ts_manager.cc:194] Registered new tserver with Master: b9dc99bb5e754aea8f7b8fef88216c14 (127.15.127.129:41913)
I20260924 07:17:09.957554 26115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:60019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:17:09.968271 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38489
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:10.086115 26311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:10.086429 26311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:10.086520 26311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:10.090793 26311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:10.090886 26311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:10.090976 26311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:17:10.096516 26311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:10.097744 26311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:10.098883 26311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:10.108577 26317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:10.108551 26316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:10.108752 26311 server_base.cc:1034] running on GCE node
W20260924 07:17:10.108963 26319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:10.109418 26311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:10.109989 26311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:10.111153 26311 hybrid_clock.cc:648] HybridClock initialized: now 1790234230111122 us; error 43 us; skew 500 ppm
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:10.113914 26311 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:17:10.114811 26311 webserver.cc:460] Webserver started at http://127.15.127.130:42673/ using document root <none> and password file <none>
I20260924 07:17:10.115370 26311 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:10.115429 26311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:10.115655 26311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:10.117748 26311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "48d3c08d8698491e9913c05dd9941552"
format_stamp: "Formatted at 2026-09-24 07:17:10 on dist-test-slave-zx46"
server_key: "9511473177e9c0c8f8ee0f8035fb53dc"
server_key_iv: "564216c1ca1eefedfce3479071c538e5"
server_key_version: "encryptionkey@0"
I20260924 07:17:10.118315 26311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "48d3c08d8698491e9913c05dd9941552"
format_stamp: "Formatted at 2026-09-24 07:17:10 on dist-test-slave-zx46"
server_key: "9511473177e9c0c8f8ee0f8035fb53dc"
server_key_iv: "564216c1ca1eefedfce3479071c538e5"
server_key_version: "encryptionkey@0"
I20260924 07:17:10.121995 26311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:17:10.124399 26326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:10.125674 26311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:17:10.125806 26311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "48d3c08d8698491e9913c05dd9941552"
format_stamp: "Formatted at 2026-09-24 07:17:10 on dist-test-slave-zx46"
server_key: "9511473177e9c0c8f8ee0f8035fb53dc"
server_key_iv: "564216c1ca1eefedfce3479071c538e5"
server_key_version: "encryptionkey@0"
I20260924 07:17:10.125917 26311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:10.149935 26311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:10.153424 26311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:10.153663 26311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:10.154289 26311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:10.155138 26311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:10.155208 26311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:10.155279 26311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:10.155331 26311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:10.163659 26311 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:44839
I20260924 07:17:10.163687 26439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:44839 every 8 connection(s)
I20260924 07:17:10.164543 26311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:17:10.165124 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 26311
I20260924 07:17:10.165242 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:17:10.165572 15870 external_mini_cluster.cc:1428] Setting key bf3b6d1b5dc3eae2d2c425aa1fd179f6
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:17:10.179543 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.166454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:53961 (local address 127.15.127.190:38489)
0924 07:17:10.166589 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.166593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.167578 (+ 985us) server_negotiation.cc:407] Connection header received
0924 07:17:10.169077 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.169080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.169164 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.169270 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:10.170410 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.170935 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.172069 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.172300 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.172412 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.175113 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:10.175116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:10.175120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:10.175145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:10.176821 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:10.177490 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:10.177493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:10.177494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:10.177570 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:10.178022 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:10.178029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:10.178033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:10.178308 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:10.178400 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.179188 (+ 788us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.179338 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:17:10.180688 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.166940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38489 (local address 127.15.127.130:53961)
0924 07:17:10.167438 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.167479 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.168847 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.169442 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.169448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.169811 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:10.170269 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.170282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.171057 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.171062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.171943 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.171952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.172449 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.172453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.172625 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.173276 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:10.173292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:10.174973 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:10.176926 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:10.176931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:10.176944 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:10.177396 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:10.177671 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:10.177674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:10.177676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:10.177866 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:10.178474 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:10.178483 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:10.178821 (+ 338us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.179093 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.179499 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260924 07:17:10.181914 26440 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38489
I20260924 07:17:10.182195 26440 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:10.182844 26440 heartbeater.cc:511] Master 127.15.127.190:38489 requested a full tablet report, sending...
I20260924 07:17:10.183952 26115 ts_manager.cc:194] Registered new tserver with Master: 48d3c08d8698491e9913c05dd9941552 (127.15.127.130:44839)
I20260924 07:17:10.184525 26115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:53961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:17:10.195638 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38489
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:10.309325 26445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:10.309624 26445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:10.309670 26445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:10.313948 26445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:10.314024 26445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:10.314085 26445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:17:10.319643 26445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:10.320777 26445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:10.321926 26445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:10.331954 26450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:10.331987 26453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:10.331938 26451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:10.332068 26445 server_base.cc:1034] running on GCE node
I20260924 07:17:10.332808 26445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:10.333616 26445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:10.334826 26445 hybrid_clock.cc:648] HybridClock initialized: now 1790234230334810 us; error 31 us; skew 500 ppm
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:10.341246 26445 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:17:10.342228 26445 webserver.cc:460] Webserver started at http://127.15.127.131:43227/ using document root <none> and password file <none>
I20260924 07:17:10.342841 26445 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:10.342937 26445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:10.343181 26445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:17:10.345402 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "c8b7afe84c4e4d1aa7884b37641241db"
format_stamp: "Formatted at 2026-09-24 07:17:10 on dist-test-slave-zx46"
server_key: "08c5027bc6918e41cdc62cd56f201c1a"
server_key_iv: "c304dd3e9bf2ead1e6d85dae9a28cf55"
server_key_version: "encryptionkey@0"
I20260924 07:17:10.345992 26445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "c8b7afe84c4e4d1aa7884b37641241db"
format_stamp: "Formatted at 2026-09-24 07:17:10 on dist-test-slave-zx46"
server_key: "08c5027bc6918e41cdc62cd56f201c1a"
server_key_iv: "c304dd3e9bf2ead1e6d85dae9a28cf55"
server_key_version: "encryptionkey@0"
I20260924 07:17:10.349849 26445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:17:10.352357 26460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:10.353678 26445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:17:10.353828 26445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "c8b7afe84c4e4d1aa7884b37641241db"
format_stamp: "Formatted at 2026-09-24 07:17:10 on dist-test-slave-zx46"
server_key: "08c5027bc6918e41cdc62cd56f201c1a"
server_key_iv: "c304dd3e9bf2ead1e6d85dae9a28cf55"
server_key_version: "encryptionkey@0"
I20260924 07:17:10.353946 26445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:10.367430 26445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:10.370464 26445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:10.370704 26445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:10.371317 26445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:10.372164 26445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:10.372236 26445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:10.372309 26445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:10.372361 26445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:10.382025 26445 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:37365
I20260924 07:17:10.382043 26573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:37365 every 8 connection(s)
I20260924 07:17:10.383001 26445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:17:10.392333 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 26445
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:17:10.392494 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:17:10.392840 15870 external_mini_cluster.cc:1428] Setting key 22ef2851ecbba46be7ec06ff450a3630
I20260924 07:17:10.396726 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.384770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:59853 (local address 127.15.127.190:38489)
0924 07:17:10.384909 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.384913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.385820 (+ 907us) server_negotiation.cc:407] Connection header received
0924 07:17:10.387161 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.387164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.387210 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.387309 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:10.388471 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.388989 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.390238 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.390464 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.390572 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.393139 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:10.393143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:10.393146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:10.393172 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:10.394799 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:10.395364 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:10.395367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:10.395368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:10.395439 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:10.395755 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:10.395758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:10.395759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:10.395929 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:10.396005 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.396496 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.396599 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:17:10.397616 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.385228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38489 (local address 127.15.127.131:59853)
0924 07:17:10.385661 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.385704 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.386933 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.387472 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.387479 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.387832 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:10.388337 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.388350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.389194 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.389197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.390085 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.390094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.390622 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.390626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.390798 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.391400 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:10.391415 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:10.392908 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:10.394942 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:10.394948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:10.394963 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:10.395278 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:10.395517 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:10.395519 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:10.395521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:10.395669 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:10.396043 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:10.396049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:10.396289 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.396451 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.396712 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":110,"threads_started":1}
I20260924 07:17:10.398736 26574 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38489
I20260924 07:17:10.398984 26574 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:10.399528 26574 heartbeater.cc:511] Master 127.15.127.190:38489 requested a full tablet report, sending...
I20260924 07:17:10.400524 26115 ts_manager.cc:194] Registered new tserver with Master: c8b7afe84c4e4d1aa7884b37641241db (127.15.127.131:37365)
I20260924 07:17:10.401150 26115 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:59853
I20260924 07:17:10.406682 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:17:10.416143 26154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.408644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.15.127.190:38489)
0924 07:17:10.408769 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.408772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.408853 (+ 81us) server_negotiation.cc:407] Connection header received
0924 07:17:10.408952 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.408955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.409083 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.409157 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:10.409813 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.410327 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.411141 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.411354 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.411436 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.412937 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:10.412940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:10.412943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:10.412973 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:10.414582 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:10.414941 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:10.414943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:10.414944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:10.415051 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:10.415340 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:10.415343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:10.415344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:10.415593 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:10.415700 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.415883 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.416020 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:17:10.419103 26115 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:17:10.422194 26115 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:17:10.437291 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.431025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.15.127.131:37365)
0924 07:17:10.431792 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.431805 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.431829 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:17:10.431907 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.431913 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.432150 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.432749 (+ 599us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:10.432905 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.434013 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.435575 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.436466 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436595 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.436742 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.437057 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.437155 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":161,"threads_started":1}
I20260924 07:17:10.437255 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.430896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:37365 (local address 127.0.0.1:58614)
0924 07:17:10.431319 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.431374 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.431497 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.432341 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.432350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.432375 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:10.432556 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.432564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.434141 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.434147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.435439 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.435449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.436607 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.436747 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.436793 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.436902 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.437059 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260924 07:17:10.437279 26586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.430933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:41913 (local address 127.0.0.1:44408)
0924 07:17:10.432340 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.432354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.432482 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.433043 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.433047 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.433067 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:10.433184 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.433190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.434626 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.434630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.435754 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.435763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.436813 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.436906 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.436928 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.437045 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.437132 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":86,"threads_started":1}
I20260924 07:17:10.437659 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.431683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:44839 (local address 127.0.0.1:46622)
0924 07:17:10.432766 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.432779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.432864 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.433406 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.433409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.433427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:10.433501 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.433506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.434841 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.434844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.435902 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.435910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.436965 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.437094 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.437115 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.437185 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.437246 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"thread_start_us":97,"threads_started":1}
I20260924 07:17:10.437855 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.431863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.15.127.130:44839)
0924 07:17:10.432268 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.432281 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.432967 (+ 686us) server_negotiation.cc:407] Connection header received
0924 07:17:10.433066 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.433071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.433266 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.433418 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:10.433628 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.434704 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436023 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.436655 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436783 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.436908 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.437365 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.437713 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":103,"threads_started":1}
I20260924 07:17:10.438139 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.431787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44408 (local address 127.15.127.129:41913)
0924 07:17:10.432040 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.432056 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.432465 (+ 409us) server_negotiation.cc:407] Connection header received
0924 07:17:10.432635 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.432641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.432865 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.433051 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:10.433291 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.434464 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.435890 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.436675 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.436825 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.436942 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.437907 (+ 965us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.438007 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260924 07:17:10.440431 26508 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf5ab5a2fc649d49457e338ee232d2b (DEFAULT_TABLE table=test-table [id=6100f7c78baa4692b55423dbbc7d72de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:17:10.440857 26374 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf5ab5a2fc649d49457e338ee232d2b (DEFAULT_TABLE table=test-table [id=6100f7c78baa4692b55423dbbc7d72de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:17:10.440857 26240 tablet_service.cc:1467] Processing CreateTablet for tablet 5cf5ab5a2fc649d49457e338ee232d2b (DEFAULT_TABLE table=test-table [id=6100f7c78baa4692b55423dbbc7d72de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:17:10.441576 26508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf5ab5a2fc649d49457e338ee232d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:10.441890 26374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf5ab5a2fc649d49457e338ee232d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:10.441897 26240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cf5ab5a2fc649d49457e338ee232d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:10.447664 26591 tablet_bootstrap.cc:492] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Bootstrap starting.
I20260924 07:17:10.449199 26593 tablet_bootstrap.cc:492] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14: Bootstrap starting.
I20260924 07:17:10.449721 26592 tablet_bootstrap.cc:492] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: Bootstrap starting.
I20260924 07:17:10.450801 26591 tablet_bootstrap.cc:654] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:10.451375 26593 tablet_bootstrap.cc:654] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:10.451974 26592 tablet_bootstrap.cc:654] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:10.452212 26591 log.cc:824] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:10.452602 26593 log.cc:824] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:10.452736 26592 log.cc:824] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:10.454370 26591 tablet_bootstrap.cc:492] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: No bootstrap required, opened a new log
I20260924 07:17:10.454500 26592 tablet_bootstrap.cc:492] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: No bootstrap required, opened a new log
I20260924 07:17:10.454576 26591 ts_tablet_manager.cc:1397] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 07:17:10.454707 26592 ts_tablet_manager.cc:1397] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:17:10.457993 26591 raft_consensus.cc:348] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.457978 26592 raft_consensus.cc:348] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.458221 26592 raft_consensus.cc:374] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:10.458205 26591 raft_consensus.cc:374] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:10.458302 26591 raft_consensus.cc:729] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8b7afe84c4e4d1aa7884b37641241db, State: Initialized, Role: FOLLOWER
I20260924 07:17:10.458297 26592 raft_consensus.cc:729] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48d3c08d8698491e9913c05dd9941552, State: Initialized, Role: FOLLOWER
I20260924 07:17:10.458782 26591 consensus_queue.cc:260] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.458792 26592 consensus_queue.cc:260] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.459450 26574 heartbeater.cc:503] Master 127.15.127.190:38489 was elected leader, sending a full tablet report...
I20260924 07:17:10.459681 26440 heartbeater.cc:503] Master 127.15.127.190:38489 was elected leader, sending a full tablet report...
I20260924 07:17:10.459744 26592 ts_tablet_manager.cc:1428] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:17:10.459745 26591 ts_tablet_manager.cc:1428] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:17:10.461551 26593 tablet_bootstrap.cc:492] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14: No bootstrap required, opened a new log
I20260924 07:17:10.461767 26593 ts_tablet_manager.cc:1397] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20260924 07:17:10.465456 26593 raft_consensus.cc:348] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.465678 26593 raft_consensus.cc:374] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:10.465751 26593 raft_consensus.cc:729] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9dc99bb5e754aea8f7b8fef88216c14, State: Initialized, Role: FOLLOWER
I20260924 07:17:10.466249 26593 consensus_queue.cc:260] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.466889 26306 heartbeater.cc:503] Master 127.15.127.190:38489 was elected leader, sending a full tablet report...
I20260924 07:17:10.467229 26593 ts_tablet_manager.cc:1428] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:17:10.571262 26598 raft_consensus.cc:482] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:17:10.571486 26598 raft_consensus.cc:504] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.572616 26598 leader_election.cc:288] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9dc99bb5e754aea8f7b8fef88216c14 (127.15.127.129:41913), 48d3c08d8698491e9913c05dd9941552 (127.15.127.130:44839)
I20260924 07:17:10.576359 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.572922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:41913 (local address 127.15.127.131:38755)
0924 07:17:10.573092 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.573112 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.573216 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.573562 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.573565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.573581 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:10.573718 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.573723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.574456 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.574463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.575274 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.575282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.575923 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.575926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.576029 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.576051 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.576152 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.576227 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:17:10.576359 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.573202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:38755 (local address 127.15.127.129:41913)
0924 07:17:10.573360 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.573364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.573376 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:17:10.573422 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.573425 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.573460 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.573549 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:10.573829 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.574343 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.575407 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.575804 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.575931 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.575987 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.576167 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.576238 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:17:10.576632 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.573037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:50485 (local address 127.15.127.130:44839)
0924 07:17:10.573176 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.573180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.573640 (+ 460us) server_negotiation.cc:407] Connection header received
0924 07:17:10.573794 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.573797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.573845 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.573923 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:10.574208 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.574693 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.575716 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.576113 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.576239 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.576287 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.576457 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.576523 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:17:10.576648 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.572922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:44839 (local address 127.15.127.131:50485)
0924 07:17:10.573540 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.573552 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:10.573679 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:10.573942 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:10.573945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:10.573959 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:10.574082 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.574087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.574799 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.574805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.575589 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:10.575597 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.576230 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.576234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:10.576329 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.576357 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 07:17:10.576443 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:17:10.576509 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":273,"thread_start_us":148,"threads_started":1}
I20260924 07:17:10.577118 26260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf5ab5a2fc649d49457e338ee232d2b" candidate_uuid: "c8b7afe84c4e4d1aa7884b37641241db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" is_pre_election: true
I20260924 07:17:10.577374 26394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf5ab5a2fc649d49457e338ee232d2b" candidate_uuid: "c8b7afe84c4e4d1aa7884b37641241db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d3c08d8698491e9913c05dd9941552" is_pre_election: true
I20260924 07:17:10.577494 26260 raft_consensus.cc:2454] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b7afe84c4e4d1aa7884b37641241db in term 0.
I20260924 07:17:10.577697 26394 raft_consensus.cc:2454] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8b7afe84c4e4d1aa7884b37641241db in term 0.
I20260924 07:17:10.577991 26461 leader_election.cc:302] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9dc99bb5e754aea8f7b8fef88216c14, c8b7afe84c4e4d1aa7884b37641241db; no voters:
I20260924 07:17:10.578341 26598 raft_consensus.cc:2781] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:17:10.578426 26598 raft_consensus.cc:482] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:17:10.578469 26598 raft_consensus.cc:3037] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:10.579685 26598 raft_consensus.cc:504] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.580035 26598 leader_election.cc:288] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [CANDIDATE]: Term 1 election: Requested vote from peers b9dc99bb5e754aea8f7b8fef88216c14 (127.15.127.129:41913), 48d3c08d8698491e9913c05dd9941552 (127.15.127.130:44839)
I20260924 07:17:10.580366 26260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf5ab5a2fc649d49457e338ee232d2b" candidate_uuid: "c8b7afe84c4e4d1aa7884b37641241db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc99bb5e754aea8f7b8fef88216c14"
I20260924 07:17:10.580415 26394 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cf5ab5a2fc649d49457e338ee232d2b" candidate_uuid: "c8b7afe84c4e4d1aa7884b37641241db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48d3c08d8698491e9913c05dd9941552"
I20260924 07:17:10.580505 26260 raft_consensus.cc:3037] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:10.580540 26394 raft_consensus.cc:3037] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:10.581681 26394 raft_consensus.cc:2454] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b7afe84c4e4d1aa7884b37641241db in term 1.
I20260924 07:17:10.581748 26260 raft_consensus.cc:2454] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8b7afe84c4e4d1aa7884b37641241db in term 1.
I20260924 07:17:10.581991 26464 leader_election.cc:302] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48d3c08d8698491e9913c05dd9941552, c8b7afe84c4e4d1aa7884b37641241db; no voters:
I20260924 07:17:10.582207 26598 raft_consensus.cc:2781] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:17:10.582442 26598 raft_consensus.cc:686] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 1 LEADER]: Becoming Leader. State: Replica: c8b7afe84c4e4d1aa7884b37641241db, State: Running, Role: LEADER
I20260924 07:17:10.582777 26598 consensus_queue.cc:237] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } }
I20260924 07:17:10.585937 26115 catalog_manager.cc:5495] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db reported cstate change: term changed from 0 to 1, leader changed from <none> to c8b7afe84c4e4d1aa7884b37641241db (127.15.127.131). New cstate: current_term: 1 leader_uuid: "c8b7afe84c4e4d1aa7884b37641241db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8b7afe84c4e4d1aa7884b37641241db" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20260924 07:17:10.629814 26587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.626246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.15.127.131:37365)
0924 07:17:10.626368 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:10.626372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:10.626385 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:17:10.626486 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:10.626488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:10.626554 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:10.626644 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:17:10.626883 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.627437 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.628284 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:10.628503 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:10.628592 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:10.628929 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:17:10.629455 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:17:10.629553 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:10.629619 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:17:10.629675 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 07:17:10.633891 26575 tablet.cc:2369] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:17:10.636296 26260 raft_consensus.cc:1261] T 5cf5ab5a2fc649d49457e338ee232d2b P b9dc99bb5e754aea8f7b8fef88216c14 [term 1 FOLLOWER]: Refusing update from remote peer c8b7afe84c4e4d1aa7884b37641241db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:17:10.636296 26394 raft_consensus.cc:1261] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 1 FOLLOWER]: Refusing update from remote peer c8b7afe84c4e4d1aa7884b37641241db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:17:10.637157 26598 consensus_queue.cc:1035] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [LEADER]: Connected to new peer: Peer: permanent_uuid: "48d3c08d8698491e9913c05dd9941552" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:17:10.637356 26601 consensus_queue.cc:1035] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9dc99bb5e754aea8f7b8fef88216c14" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:17:10.644479 26605 mvcc.cc:205] Tried to move back new op lower bound from 7332799408678821888 to 7332799408469635072. Current Snapshot: MvccSnapshot[applied={T|T < 7332799408678821888}]
I20260924 07:17:10.653368 26113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:17:10.653554 26113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:17:10.656275 26113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 542335cb9cae473cac080b2aeed4413c: Sending DeleteTablet for 3 replicas of tablet 5cf5ab5a2fc649d49457e338ee232d2b
I20260924 07:17:10.657276 26508 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf5ab5a2fc649d49457e338ee232d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:17:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:58614
I20260924 07:17:10.657290 26240 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf5ab5a2fc649d49457e338ee232d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:17:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:44408
I20260924 07:17:10.658133 26614 tablet_replica.cc:330] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: stopping tablet replica
I20260924 07:17:10.658488 26614 raft_consensus.cc:2229] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:17:10.658511 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 26177
I20260924 07:17:10.658892 26614 raft_consensus.cc:2258] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:17:10.659551 26374 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cf5ab5a2fc649d49457e338ee232d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:17:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:46622
I20260924 07:17:10.660429 26614 ts_tablet_manager.cc:1905] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:17:10.663205 26614 ts_tablet_manager.cc:1918] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:17:10.663311 26614 log.cc:1196] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/5cf5ab5a2fc649d49457e338ee232d2b
I20260924 07:17:10.663633 26614 ts_tablet_manager.cc:1939] T 5cf5ab5a2fc649d49457e338ee232d2b P c8b7afe84c4e4d1aa7884b37641241db: Deleting consensus metadata
I20260924 07:17:10.664784 26102 catalog_manager.cc:4841] TS c8b7afe84c4e4d1aa7884b37641241db (127.15.127.131:37365): tablet 5cf5ab5a2fc649d49457e338ee232d2b (table test-table [id=6100f7c78baa4692b55423dbbc7d72de]) successfully deleted
I20260924 07:17:10.669265 26615 tablet_replica.cc:330] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552: stopping tablet replica
I20260924 07:17:10.669689 26615 raft_consensus.cc:2229] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:17:10.670109 26615 raft_consensus.cc:2258] T 5cf5ab5a2fc649d49457e338ee232d2b P 48d3c08d8698491e9913c05dd9941552 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:17:10.670734 26099 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:41913 (error 108)
I20260924 07:17:10.671556 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 26311
I20260924 07:17:10.673750 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.673419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:41913 (local address 127.0.0.1:44422)
0924 07:17:10.673599 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.673668 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:41913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260924 07:17:10.673902 26099 catalog_manager.cc:4584] TS b9dc99bb5e754aea8f7b8fef88216c14 (127.15.127.129:41913): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5cf5ab5a2fc649d49457e338ee232d2b: Network error: Client connection negotiation failed: client connection to 127.15.127.129:41913: connect: Connection refused (error 111)
I20260924 07:17:10.677884 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 26445
I20260924 07:17:10.678499 26588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:10.678166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:44839 (local address 127.0.0.1:46626)
0924 07:17:10.678342 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:10.678402 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:44839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260924 07:17:10.683905 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 26083
2026-09-24T07:17:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2376 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26619](info): set up 2 sockets
Sep 24 07:17:10 dist-test-slave-zx46 krb5kdc[26619](info): commencing operation
krb5kdc: starting...
W20260924 07:17:11.750017 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:17:11 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:17:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:17:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/rangeradmin_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/rangeradmin_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/rangerlookup_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/rangerlookup_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
I20260924 07:17:11.918079 15870 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPYFHG9/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/postgres -l logfile start
2026-09-24 07:17:14.455 UTC [26649] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:17:14.456 UTC [26649] LOG: listening on IPv4 address "127.15.127.148", port 47611
2026-09-24 07:17:14.459 UTC [26649] LOG: listening on Unix socket "/tmp/.s.PGSQL.47611"
2026-09-24 07:17:14.496 UTC [26652] LOG: database system was shut down at 2026-09-24 07:17:12 UTC
2026-09-24 07:17:14.501 UTC [26649] LOG: database system is ready to accept connections
I20260924 07:17:15.449744 15870 mini_postgres.cc:96] Postgres bound to 47611
2026-09-24 07:17:15.456 UTC [26660] FATAL: database "slave" does not exist
127.15.127.148:47611 - accepting connections
I20260924 07:17:15.456713 15870 mini_ranger.cc:161] Starting Ranger...
I20260924 07:17:15.471236 15870 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:17:15.958864 15870 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:17:15.958966 15870 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-admin
2026-09-24 07:17:16,237 [I] DB FLAVOR :POSTGRES
2026-09-24 07:17:16,238 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:17:16,238 [I] Checking connection..
2026-09-24 07:17:16,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:17:16,513 [I] Checking connection passed.
2026-09-24 07:17:16,514 [I] --------- Verifying version history table ---------
2026-09-24 07:17:16,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:17:16,774 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:17:16,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:17:17,020 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:17:17,020 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:17:17,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:17:17.259 UTC [26724] WARNING: there is no transaction in progress
2026-09-24 07:17:17,265 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:17:17,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:17:17,502 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:17:17,503 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:17:17,503 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:17:17,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:17:17,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zx46.c.gcp-upstream.internal','N') ;"
2026-09-24 07:17:18,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:17:18,457 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:17:18,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:17:18,685 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:17:18,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:17:18,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:17:19,161 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:17:19,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:17:20.120 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.136 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.152 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.167 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.183 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.200 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.263 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.270 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.284 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.293 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.303 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.316 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.326 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.332 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.339 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20.345 UTC [26832] WARNING: there is no transaction in progress
2026-09-24 07:17:20,964 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:17:20,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:17:21,218 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:17:21,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:17:21,472 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:17:21,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:17:21,732 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:17:21,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:17:21,990 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:17:22,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zx46.c.gcp-upstream.internal';"
2026-09-24 07:17:22,468 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:17:22,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:17:22,719 [I] DB_PATCHES have already been applied
I20260924 07:17:22.725356 15870 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPYFHG9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:17:22.725421 15870 mini_ranger.cc:193] Using host: 127.15.127.148
I20260924 07:17:22.728874 15870 mini_ranger.cc:227] Ranger admin URL: http://127.15.127.148:44997
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:17:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:17:23.838382 15870 mini_ranger.cc:160] Time spent starting Ranger: real 8.382s user 0.001s sys 0.009s
Sep 24, 2026 7:17:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:17:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 7:17:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:17:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:17:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:17:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:17:44.528196 15870 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/rangerkms_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/rangerkms_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.127.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.127.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:17:44 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234264, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:17:44.623669 15870 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:17:44.634014 15870 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:17:45.491909 15870 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:17:45.492009 15870 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms
I20260924 07:17:45.492113 15870 mini_ranger_kms.cc:227] Using postgres at 127.15.127.148:47611
2026-09-24 07:17:45,676 [I] DB FLAVOR :POSTGRES
2026-09-24 07:17:45,676 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:17:45,676 [I] Checking connection
2026-09-24 07:17:45,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:17:45,915 [I] connection success
2026-09-24 07:17:45,915 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:17:45,915 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:17:45,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:17:46,163 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:17:46,163 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:17:46,164 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:17:46,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:47611/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:17:46,421 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:17:46.894313 15870 mini_ranger_kms.cc:310] Created kms service
I20260924 07:17:47.080991 15870 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:17:47.123888 15870 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:17:47 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234264, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:17:47.502080 15870 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:17:47.502240 15870 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:17:47.502267 15870 mini_ranger_kms.cc:240] Using host: 127.15.127.148
I20260924 07:17:47.506593 15870 mini_ranger_kms.cc:276] Ranger KMS PID: 27012
I20260924 07:17:47.506724 15870 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.127.148:47599
Sep 24, 2026 7:17:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:17:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:17:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:17:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:17:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:17:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 07:17:49 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.127.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:49.830036 15870 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.206s user 0.009s sys 0.154s
I20260924 07:17:49.830159 15870 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:17:49.830256 15870 mini_ranger_kms.cc:401] 127.15.127.148:47599/kms/v1/keys
Sep 24 07:17:50 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.127.148@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
Sep 24 07:17:50 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234269, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.127.148@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
Sep 24 07:17:52 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:17:52.840095 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44551
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:44551
--ranger_config_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:47599/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:52.956782 27063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:52.957170 27063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:52.957271 27063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:52.961400 27063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:17:52.961495 27063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:52.961534 27063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:52.961571 27063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:17:52.961608 27063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:17:52.967502 27063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.127.148:47599/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:44551
--ranger_config_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:52.968673 27063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:52.969817 27063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:52.979352 27071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:52.979349 27068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:52.979656 27063 server_base.cc:1034] running on GCE node
W20260924 07:17:52.979491 27069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:52.980286 27063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:52.981644 27063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:52.982842 27063 hybrid_clock.cc:648] HybridClock initialized: now 1790234272982820 us; error 32 us; skew 500 ppm
Sep 24 07:17:52 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234272, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:52.989579 27063 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:17:52.990556 27063 webserver.cc:460] Webserver started at http://127.15.127.190:45033/ using document root <none> and password file <none>
I20260924 07:17:52.991123 27063 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:52.991202 27063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:52.991428 27063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:17:52 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234272, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:17:53.046867 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "bb889eb74b0a48af8658697042e710b9"
format_stamp: "Formatted at 2026-09-24 07:17:53 on dist-test-slave-zx46"
server_key: "20989c6bc149c1178feaa4ce5046a11e"
server_key_iv: "8257c14d51e74682afa03077be4008f8"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:53.047539 27063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "bb889eb74b0a48af8658697042e710b9"
format_stamp: "Formatted at 2026-09-24 07:17:53 on dist-test-slave-zx46"
server_key: "20989c6bc149c1178feaa4ce5046a11e"
server_key_iv: "8257c14d51e74682afa03077be4008f8"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:53.051612 27063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 07:17:53.066200 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:53.067927 27063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:17:53.068084 27063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "bb889eb74b0a48af8658697042e710b9"
format_stamp: "Formatted at 2026-09-24 07:17:53 on dist-test-slave-zx46"
server_key: "20989c6bc149c1178feaa4ce5046a11e"
server_key_iv: "8257c14d51e74682afa03077be4008f8"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:53.068228 27063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:53.095681 27063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:53.098748 27063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:53.098960 27063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:53.107195 27063 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:44551
I20260924 07:17:53.107249 27130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:44551 every 8 connection(s)
I20260924 07:17:53.108289 27063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:17:53.111195 27131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:53.116856 27131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9: Bootstrap starting.
I20260924 07:17:53.118379 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 27063
I20260924 07:17:53.118517 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:17:53.118824 15870 external_mini_cluster.cc:1428] Setting key 0ab2b641eb63eb3da5c08ee47a6c8b34
I20260924 07:17:53.119601 27131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:53.120581 27131 log.cc:824] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:53.122843 27131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9: No bootstrap required, opened a new log
I20260924 07:17:53.127631 27131 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb889eb74b0a48af8658697042e710b9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44551 } }
I20260924 07:17:53.127938 27131 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:53.128027 27131 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb889eb74b0a48af8658697042e710b9, State: Initialized, Role: FOLLOWER
I20260924 07:17:53.128525 27131 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb889eb74b0a48af8658697042e710b9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44551 } }
I20260924 07:17:53.128686 27131 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:17:53.128768 27131 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:17:53.128901 27131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:17:53 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:17:53.130059 27131 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb889eb74b0a48af8658697042e710b9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44551 } }
I20260924 07:17:53.130453 27131 leader_election.cc:302] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb889eb74b0a48af8658697042e710b9; no voters:
I20260924 07:17:53.130859 27131 leader_election.cc:288] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:17:53.131121 27136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:17:53.131659 27136 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [term 1 LEADER]: Becoming Leader. State: Replica: bb889eb74b0a48af8658697042e710b9, State: Running, Role: LEADER
I20260924 07:17:53.132035 27136 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb889eb74b0a48af8658697042e710b9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44551 } }
I20260924 07:17:53.132315 27131 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:17:53.134784 27131 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:17:53.136286 27134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:53.120048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.15.127.190:44551)
0924 07:17:53.120545 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:53.120564 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:53.120609 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:17:53.121393 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:53.121414 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:53.121844 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:53.122214 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:53.123060 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:53.124936 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:53.126122 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:53.126834 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:53.127013 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:53.130293 (+ 3280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:53.130299 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:53.130317 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:53.130349 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:53.132451 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:53.132938 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:53.132944 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:53.132951 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:53.133128 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:53.133422 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:53.133427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:53.133432 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:53.134014 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:53.134226 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:53.135019 (+ 793us) server_negotiation.cc:299] Negotiation successful
0924 07:17:53.135301 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":125,"threads_started":1}
I20260924 07:17:53.134750 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb889eb74b0a48af8658697042e710b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb889eb74b0a48af8658697042e710b9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44551 } } }
I20260924 07:17:53.139715 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:53.141384 27138 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb889eb74b0a48af8658697042e710b9. Latest consensus state: current_term: 1 leader_uuid: "bb889eb74b0a48af8658697042e710b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb889eb74b0a48af8658697042e710b9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44551 } } }
I20260924 07:17:53.141551 27138 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:17:53.142091 27153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:17:53.145511 27153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:17:53.152776 27153 catalog_manager.cc:1347] Generated new cluster ID: 04d559b9d28644b39a446d0ee14eac42
I20260924 07:17:53.152901 27153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:17:53.169795 27153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:17:53.171614 27153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:17:53.186771 27153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9: Generated new TSK 0
I20260924 07:17:53.187884 27153 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:17:53.919817 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:47599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:44551
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:54.046293 27176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:54.046666 27176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:54.046738 27176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:54.052929 27176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:54.053061 27176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:54.053164 27176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:17:54.060015 27176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.127.148:47599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:44551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:54.061553 27176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:54.062613 27176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:54.073103 27183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:54.073120 27184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:54.074275 27186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:54.074867 27176 server_base.cc:1034] running on GCE node
I20260924 07:17:54.075430 27176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:54.076124 27176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:54.077296 27176 hybrid_clock.cc:648] HybridClock initialized: now 1790234274077266 us; error 48 us; skew 500 ppm
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:54.080271 27176 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:17:54.081416 27176 webserver.cc:460] Webserver started at http://127.15.127.129:36629/ using document root <none> and password file <none>
I20260924 07:17:54.082028 27176 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:54.082122 27176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:54.082365 27176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:17:54.100450 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "8ff81e03f50e4af4bf4b01d86252fc6e"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "03536ce84914bbf5aae47f71a5414c70"
server_key_iv: "9f4d1ef5b56a92fc828f602b1210bccf"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.101750 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "8ff81e03f50e4af4bf4b01d86252fc6e"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "03536ce84914bbf5aae47f71a5414c70"
server_key_iv: "9f4d1ef5b56a92fc828f602b1210bccf"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.107190 27176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 07:17:54.131389 27193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.133137 27176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:17:54.133270 27176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "8ff81e03f50e4af4bf4b01d86252fc6e"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "03536ce84914bbf5aae47f71a5414c70"
server_key_iv: "9f4d1ef5b56a92fc828f602b1210bccf"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.133422 27176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:54.155236 27176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:54.174175 27176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:54.174412 27176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:54.175232 27176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:54.176467 27176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:54.176553 27176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.176616 27176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:54.176651 27176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.187764 27176 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:43657
I20260924 07:17:54.189851 27176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:17:54.192094 27308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:43657 every 8 connection(s)
I20260924 07:17:54.195420 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 27176
I20260924 07:17:54.195555 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:17:54.195880 15870 external_mini_cluster.cc:1428] Setting key 297946c2633e91df80ce555b8f6b665a
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:17:54.234460 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:54.195523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:44485 (local address 127.15.127.190:44551)
0924 07:17:54.197451 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:54.197456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:54.197871 (+ 415us) server_negotiation.cc:407] Connection header received
0924 07:17:54.198777 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:54.198780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:54.198831 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:54.198918 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:54.212806 (+ 13888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.213768 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.215587 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.216330 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.216449 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:54.229675 (+ 13226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:54.229680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:54.229683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:54.229720 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:54.231662 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:54.232356 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:54.232360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:54.232362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:54.232481 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:54.232926 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:54.232930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:54.232932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:54.233222 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:54.233316 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:54.234118 (+ 802us) server_negotiation.cc:299] Negotiation successful
0924 07:17:54.234256 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":78,"threads_started":1}
I20260924 07:17:54.236445 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:54.194445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:44551 (local address 127.15.127.129:44485)
0924 07:17:54.197717 (+ 3272us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:54.197772 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:54.198580 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:54.211311 (+ 12731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:54.211326 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:54.211759 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:54.212574 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:54.212595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.213959 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.213963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:54.215431 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:54.215444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.225180 (+ 9736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.225186 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:54.225482 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:54.227469 (+ 1987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:54.227500 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:54.229383 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:54.231825 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:54.231832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:54.231851 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:54.232216 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:54.232593 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:54.232597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:54.232599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:54.232803 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:54.233380 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:54.233387 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:54.233762 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 07:17:54.234952 (+ 1190us) client_negotiation.cc:240] Negotiation successful
0924 07:17:54.235333 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3059,"thread_start_us":124,"threads_started":1}
I20260924 07:17:54.238878 27309 heartbeater.cc:347] Connected to a master server at 127.15.127.190:44551
I20260924 07:17:54.239135 27309 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:54.240172 27309 heartbeater.cc:511] Master 127.15.127.190:44551 requested a full tablet report, sending...
I20260924 07:17:54.242127 27095 ts_manager.cc:194] Registered new tserver with Master: 8ff81e03f50e4af4bf4b01d86252fc6e (127.15.127.129:43657)
I20260924 07:17:54.243760 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:44485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:17:54.270010 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:47599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:44551
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:17:54.449857 27315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:54.450316 27315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:54.450464 27315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:54.456845 27315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:54.457026 27315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:54.457197 27315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:17:54.465456 27315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.127.148:47599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:44551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:54.467167 27315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:54.468632 27315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:17:54.483181 27324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:54.483520 27322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:54.487545 27315 server_base.cc:1034] running on GCE node
W20260924 07:17:54.489161 27321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:54.489768 27315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:54.490629 27315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:54.491825 27315 hybrid_clock.cc:648] HybridClock initialized: now 1790234274491755 us; error 92 us; skew 500 ppm
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:54.495609 27315 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:17:54.496799 27315 webserver.cc:460] Webserver started at http://127.15.127.130:33677/ using document root <none> and password file <none>
I20260924 07:17:54.497638 27315 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:54.497799 27315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:54.498102 27315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:17:54.514966 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "6976a9722563476eaffd3601df9f9089"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "7148e34edc8b2e3a54c9a79dc8498e70"
server_key_iv: "d153884e3c7e2b54f7e63b5364bde6a6"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.515885 27315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "6976a9722563476eaffd3601df9f9089"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "7148e34edc8b2e3a54c9a79dc8498e70"
server_key_iv: "d153884e3c7e2b54f7e63b5364bde6a6"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.526799 27315 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
I20260924 07:17:54.551286 27331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.553236 27315 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260924 07:17:54.553385 27315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "6976a9722563476eaffd3601df9f9089"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "7148e34edc8b2e3a54c9a79dc8498e70"
server_key_iv: "d153884e3c7e2b54f7e63b5364bde6a6"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.553530 27315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:54.581548 27315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:54.585583 27315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:54.585860 27315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:54.586691 27315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:54.587927 27315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:54.588053 27315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.588160 27315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:54.588233 27315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.610522 27315 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:46107
I20260924 07:17:54.611783 27315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:17:54.613945 27444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:46107 every 8 connection(s)
I20260924 07:17:54.618121 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 27315
I20260924 07:17:54.618283 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:17:54.618640 15870 external_mini_cluster.cc:1428] Setting key 5b62c964f6a104107ee38db7e263a45a
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:17:54.631678 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:54.616010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:55461 (local address 127.15.127.190:44551)
0924 07:17:54.616152 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:54.616157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:54.617268 (+ 1111us) server_negotiation.cc:407] Connection header received
0924 07:17:54.618560 (+ 1292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:54.618565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:54.618626 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:54.618723 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:54.620298 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.621116 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.622979 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.623380 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.623473 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:54.626864 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:54.626870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:54.626875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:54.626909 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:54.628853 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:54.629616 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:54.629620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:54.629622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:54.629714 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:54.630170 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:54.630173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:54.630175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:54.630439 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:54.630549 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:54.631373 (+ 824us) server_negotiation.cc:299] Negotiation successful
0924 07:17:54.631510 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:17:54.632789 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:54.616461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:44551 (local address 127.15.127.130:55461)
0924 07:17:54.617101 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:54.617159 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:54.618281 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:54.619042 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:54.619053 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:54.619441 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:54.620132 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:54.620153 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.621293 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.621298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:54.622829 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:54.622842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.623701 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.623707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:54.623947 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:54.624696 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:54.624723 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:54.626623 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:54.629037 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:54.629047 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:54.629070 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:54.629483 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:54.629852 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:54.629857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:54.629859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:54.630061 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:54.630820 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:54.630826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:54.631099 (+ 273us) client_negotiation.cc:769] Sending connection context
0924 07:17:54.631322 (+ 223us) client_negotiation.cc:240] Negotiation successful
0924 07:17:54.631682 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":161,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:17:54.634184 27445 heartbeater.cc:347] Connected to a master server at 127.15.127.190:44551
I20260924 07:17:54.634476 27445 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:54.634990 27445 heartbeater.cc:511] Master 127.15.127.190:44551 requested a full tablet report, sending...
I20260924 07:17:54.636093 27095 ts_manager.cc:194] Registered new tserver with Master: 6976a9722563476eaffd3601df9f9089 (127.15.127.130:46107)
I20260924 07:17:54.636937 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:55461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:17:54.663199 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:47599/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:44551
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:46097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:17:54.805166 27450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:17:54.805598 27450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:17:54.805730 27450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:17:54.811846 27450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:17:54.812002 27450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:17:54.812142 27450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:17:54.820314 27450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:46097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.127.148:47599/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:44551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:17:54.821975 27450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:17:54.823366 27450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
W20260924 07:17:54.837126 27456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:54.837316 27458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:17:54.837862 27455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:17:54.846494 27450 server_base.cc:1034] running on GCE node
I20260924 07:17:54.847038 27450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:17:54.847715 27450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:17:54.848891 27450 hybrid_clock.cc:648] HybridClock initialized: now 1790234274848814 us; error 94 us; skew 500 ppm
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:17:54.852424 27450 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:17:54.853645 27450 webserver.cc:460] Webserver started at http://127.15.127.131:33279/ using document root <none> and password file <none>
I20260924 07:17:54.854382 27450 fs_manager.cc:359] Metadata directory not provided
I20260924 07:17:54.854482 27450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:17:54.854732 27450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:17:54.873145 27450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "4ceabe26c7c341248c91c57e5766cf2c"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "27cd1c24c6c8c782f4b115604fdb4bc2"
server_key_iv: "3e180e573934ccd66dd9f1470bda134e"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.873924 27450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "4ceabe26c7c341248c91c57e5766cf2c"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "27cd1c24c6c8c782f4b115604fdb4bc2"
server_key_iv: "3e180e573934ccd66dd9f1470bda134e"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.879261 27450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 07:17:54.913472 27465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.915261 27450 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.002s sys 0.000s
I20260924 07:17:54.915416 27450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "4ceabe26c7c341248c91c57e5766cf2c"
format_stamp: "Formatted at 2026-09-24 07:17:54 on dist-test-slave-zx46"
server_key: "27cd1c24c6c8c782f4b115604fdb4bc2"
server_key_iv: "3e180e573934ccd66dd9f1470bda134e"
server_key_version: "kuduclusterkey@0"
I20260924 07:17:54.915571 27450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:17:54.931445 27450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:17:54.941552 27450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:17:54.941774 27450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:17:54.942493 27450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:17:54.943504 27450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:17:54.943568 27450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.943625 27450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:17:54.943656 27450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:17:54.965369 27450 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:37359
I20260924 07:17:54.965515 27578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:37359 every 8 connection(s)
I20260924 07:17:54.966588 27450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:17:54.970675 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 27450
I20260924 07:17:54.970793 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:17:54.971083 15870 external_mini_cluster.cc:1428] Setting key 0de7360eece2eda8de9b3f4a65f161e8
Sep 24 07:17:54 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:17:55.001711 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:54.969193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:55401 (local address 127.15.127.190:44551)
0924 07:17:54.969341 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:54.969346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:54.978360 (+ 9014us) server_negotiation.cc:407] Connection header received
0924 07:17:54.989130 (+ 10770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:54.989134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:54.989196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:54.989249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:54.991134 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.991971 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.994065 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.994444 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.994519 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:54.997639 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:54.997643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:54.997645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:54.997671 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:54.999373 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:54.999983 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:54.999986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:54.999988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:55.000067 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:55.000449 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:55.000452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:55.000454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:55.000658 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:55.000732 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.001426 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.001548 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:17:55.003568 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:54.969455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:44551 (local address 127.15.127.131:55401)
0924 07:17:54.978187 (+ 8732us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:54.978249 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:54.979358 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:54.989797 (+ 10439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:54.989808 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:54.990211 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:17:54.990958 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:54.990978 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.992136 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.992139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:54.993929 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:54.993940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:54.994727 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:54.994731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:54.994931 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:54.995673 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:17:54.995695 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:17:54.997455 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:17:54.999520 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:54.999526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:54.999544 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:54.999870 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:55.000166 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:17:55.000169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:17:55.000171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:17:55.000337 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:17:55.000794 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:17:55.000800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:17:55.001140 (+ 340us) client_negotiation.cc:769] Sending connection context
0924 07:17:55.002163 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0924 07:17:55.002517 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8468,"thread_start_us":133,"threads_started":1}
I20260924 07:17:55.005033 27579 heartbeater.cc:347] Connected to a master server at 127.15.127.190:44551
I20260924 07:17:55.005271 27579 heartbeater.cc:464] Registering TS with master...
I20260924 07:17:55.005822 27579 heartbeater.cc:511] Master 127.15.127.190:44551 requested a full tablet report, sending...
I20260924 07:17:55.006938 27095 ts_manager.cc:194] Registered new tserver with Master: 4ceabe26c7c341248c91c57e5766cf2c (127.15.127.131:37359)
I20260924 07:17:55.007654 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:55401
I20260924 07:17:55.009670 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:17:55.042249 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.020924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.15.127.190:44551)
0924 07:17:55.021064 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.021069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.032798 (+ 11729us) server_negotiation.cc:407] Connection header received
0924 07:17:55.033085 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.033089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.033143 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.033197 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:17:55.034256 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.035073 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.036290 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.036637 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.036706 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.038565 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:17:55.038569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:17:55.038572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:17:55.038599 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:17:55.040401 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:55.040901 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:55.040904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:55.040906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:55.040986 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:17:55.041295 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:17:55.041297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:17:55.041299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:17:55.041510 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:17:55.041610 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.041963 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.042081 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:17:55.046101 27095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:17:55.050700 27095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:17:55.093436 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.078800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.15.127.130:46107)
0924 07:17:55.080794 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.080809 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.083491 (+ 2682us) server_negotiation.cc:407] Connection header received
0924 07:17:55.083577 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.083582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.083808 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.083963 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:55.088569 (+ 4606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.089981 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.091591 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.092480 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.092576 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.092714 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.093171 (+ 457us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.093280 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1898,"thread_start_us":69,"threads_started":1}
I20260924 07:17:55.094187 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.077895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:46107 (local address 127.0.0.1:55844)
0924 07:17:55.079510 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:55.083271 (+ 3761us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:55.083387 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:55.086813 (+ 3426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:55.086819 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:55.086841 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:55.088416 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.088428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.090096 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.090099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.091439 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.091448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.092805 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.092808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.092927 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.092965 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:17:55.093915 (+ 950us) client_negotiation.cc:240] Negotiation successful
0924 07:17:55.094048 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1432,"mutex_wait_us":2,"thread_start_us":105,"threads_started":1}
I20260924 07:17:55.097199 27360 tablet_service.cc:1467] Processing CreateTablet for tablet 817c88c44a8f4d278c54e51664d6df05 (DEFAULT_TABLE table=test-table [id=8abfc82448124f56a636b83de0208a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:17:55.097977 27592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.078759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:37359 (local address 127.0.0.1:59718)
0924 07:17:55.085208 (+ 6449us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:55.085222 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:55.085312 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:55.088557 (+ 3245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:55.088560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:55.088577 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:55.088660 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.088664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.090037 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.090043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.093617 (+ 3574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.093627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.097602 (+ 3975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.097605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.097681 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.097704 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:17:55.097769 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:17:55.097838 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6381,"mutex_wait_us":1064,"thread_start_us":70,"threads_started":1}
I20260924 07:17:55.098377 27360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817c88c44a8f4d278c54e51664d6df05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:55.098446 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.079541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.15.127.131:37359)
0924 07:17:55.080616 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.080627 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.085427 (+ 4800us) server_negotiation.cc:407] Connection header received
0924 07:17:55.085514 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.085519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.085722 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.085874 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:55.088752 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.089901 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.093765 (+ 3864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.094841 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.094954 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.095070 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.098193 (+ 3123us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.098301 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":145,"threads_started":1}
I20260924 07:17:55.100749 27497 tablet_service.cc:1467] Processing CreateTablet for tablet 817c88c44a8f4d278c54e51664d6df05 (DEFAULT_TABLE table=test-table [id=8abfc82448124f56a636b83de0208a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:17:55.101752 27497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817c88c44a8f4d278c54e51664d6df05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:55.123009 27597 tablet_bootstrap.cc:492] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Bootstrap starting.
I20260924 07:17:55.123730 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.078324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43657 (local address 127.0.0.1:53660)
0924 07:17:55.080937 (+ 2613us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:55.081023 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:55.081143 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:55.087016 (+ 5873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:55.087019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:55.087032 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:55.087907 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.087919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.092469 (+ 4550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.092472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.097468 (+ 4996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.097478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.123272 (+ 25794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.123276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.123410 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.123440 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 07:17:55.123497 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:17:55.123584 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2553,"thread_start_us":95,"threads_started":1}
I20260924 07:17:55.124500 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.078464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.15.127.129:43657)
0924 07:17:55.086167 (+ 7703us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.086175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.086197 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:17:55.086311 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.086316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.086512 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.086653 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:55.088453 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.091170 (+ 2717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.120438 (+ 29268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.122668 (+ 2230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.124065 (+ 1397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.124209 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.124263 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.124356 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7591,"thread_start_us":92,"threads_started":1}
I20260924 07:17:55.126461 27597 tablet_bootstrap.cc:654] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:55.127429 27597 log.cc:824] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:55.127902 27598 tablet_bootstrap.cc:492] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Bootstrap starting.
I20260924 07:17:55.128156 27233 tablet_service.cc:1467] Processing CreateTablet for tablet 817c88c44a8f4d278c54e51664d6df05 (DEFAULT_TABLE table=test-table [id=8abfc82448124f56a636b83de0208a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:17:55.129307 27233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 817c88c44a8f4d278c54e51664d6df05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:17:55.130692 27598 tablet_bootstrap.cc:654] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:55.132104 27598 log.cc:824] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:55.134503 27597 tablet_bootstrap.cc:492] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: No bootstrap required, opened a new log
I20260924 07:17:55.134841 27597 ts_tablet_manager.cc:1397] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 07:17:55.139168 27597 raft_consensus.cc:348] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.139441 27597 raft_consensus.cc:374] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:55.139487 27597 raft_consensus.cc:729] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ceabe26c7c341248c91c57e5766cf2c, State: Initialized, Role: FOLLOWER
I20260924 07:17:55.139937 27597 consensus_queue.cc:260] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.140967 27597 ts_tablet_manager.cc:1428] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:17:55.141376 27579 heartbeater.cc:503] Master 127.15.127.190:44551 was elected leader, sending a full tablet report...
I20260924 07:17:55.146715 27598 tablet_bootstrap.cc:492] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: No bootstrap required, opened a new log
I20260924 07:17:55.146997 27598 ts_tablet_manager.cc:1397] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260924 07:17:55.150935 27598 raft_consensus.cc:348] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.151294 27598 raft_consensus.cc:374] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:55.151386 27598 raft_consensus.cc:729] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6976a9722563476eaffd3601df9f9089, State: Initialized, Role: FOLLOWER
I20260924 07:17:55.151944 27598 consensus_queue.cc:260] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.153126 27598 ts_tablet_manager.cc:1428] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:17:55.155478 27445 heartbeater.cc:503] Master 127.15.127.190:44551 was elected leader, sending a full tablet report...
I20260924 07:17:55.160234 27601 tablet_bootstrap.cc:492] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: Bootstrap starting.
I20260924 07:17:55.162976 27601 tablet_bootstrap.cc:654] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:17:55.166759 27601 log.cc:824] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:17:55.170837 27601 tablet_bootstrap.cc:492] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: No bootstrap required, opened a new log
I20260924 07:17:55.171365 27601 ts_tablet_manager.cc:1397] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260924 07:17:55.177515 27601 raft_consensus.cc:348] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.177955 27601 raft_consensus.cc:374] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:17:55.179280 27601 raft_consensus.cc:729] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff81e03f50e4af4bf4b01d86252fc6e, State: Initialized, Role: FOLLOWER
I20260924 07:17:55.179905 27601 consensus_queue.cc:260] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.181147 27601 ts_tablet_manager.cc:1428] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260924 07:17:55.181581 27309 heartbeater.cc:503] Master 127.15.127.190:44551 was elected leader, sending a full tablet report...
W20260924 07:17:55.197106 27310 tablet.cc:2369] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:17:55.228193 27580 tablet.cc:2369] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:17:55 dist-test-slave-zx46 krb5kdc[26619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234274, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:17:55.263127 27605 raft_consensus.cc:482] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:17:55.263394 27605 raft_consensus.cc:504] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.264863 27605 leader_election.cc:288] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6976a9722563476eaffd3601df9f9089 (127.15.127.130:46107), 4ceabe26c7c341248c91c57e5766cf2c (127.15.127.131:37359)
I20260924 07:17:55.271492 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.265662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:34101 (local address 127.15.127.131:37359)
0924 07:17:55.265778 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.265783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.265808 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:17:55.265865 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.265868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.265914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.266009 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:55.266308 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.267063 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.269919 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.270536 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.270631 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.270685 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.271270 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.271353 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:17:55.272603 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.265266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:37359 (local address 127.15.127.129:34101)
0924 07:17:55.265603 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:55.265619 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:55.265765 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:55.266038 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:55.266041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:55.266061 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:55.266169 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.266175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.267899 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.267902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.269762 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.269773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.270798 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.270802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.270906 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.271143 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 07:17:55.272372 (+ 1229us) client_negotiation.cc:240] Negotiation successful
0924 07:17:55.272459 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":113,"threads_started":1}
I20260924 07:17:55.273494 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817c88c44a8f4d278c54e51664d6df05" candidate_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ceabe26c7c341248c91c57e5766cf2c" is_pre_election: true
I20260924 07:17:55.273872 27525 raft_consensus.cc:2454] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ff81e03f50e4af4bf4b01d86252fc6e in term 0.
I20260924 07:17:55.274593 27196 leader_election.cc:302] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ceabe26c7c341248c91c57e5766cf2c, 8ff81e03f50e4af4bf4b01d86252fc6e; no voters:
I20260924 07:17:55.274885 27605 raft_consensus.cc:2781] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:17:55.274999 27605 raft_consensus.cc:482] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:17:55.275050 27605 raft_consensus.cc:3037] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:55.276376 27605 raft_consensus.cc:504] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.276777 27605 leader_election.cc:288] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [CANDIDATE]: Term 1 election: Requested vote from peers 6976a9722563476eaffd3601df9f9089 (127.15.127.130:46107), 4ceabe26c7c341248c91c57e5766cf2c (127.15.127.131:37359)
I20260924 07:17:55.277179 27525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817c88c44a8f4d278c54e51664d6df05" candidate_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ceabe26c7c341248c91c57e5766cf2c"
I20260924 07:17:55.277303 27525 raft_consensus.cc:3037] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:55.278577 27525 raft_consensus.cc:2454] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ff81e03f50e4af4bf4b01d86252fc6e in term 1.
I20260924 07:17:55.278975 27196 leader_election.cc:302] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ceabe26c7c341248c91c57e5766cf2c, 8ff81e03f50e4af4bf4b01d86252fc6e; no voters:
I20260924 07:17:55.279165 27605 raft_consensus.cc:2781] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:17:55.279403 27605 raft_consensus.cc:686] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [term 1 LEADER]: Becoming Leader. State: Replica: 8ff81e03f50e4af4bf4b01d86252fc6e, State: Running, Role: LEADER
I20260924 07:17:55.279810 27605 consensus_queue.cc:237] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } }
I20260924 07:17:55.282624 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.266698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:59165 (local address 127.15.127.130:46107)
0924 07:17:55.267260 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.267265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.273358 (+ 6093us) server_negotiation.cc:407] Connection header received
0924 07:17:55.274150 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.274155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.274207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.274254 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:17:55.275749 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.276506 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.281343 (+ 4837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.281957 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.282052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.282108 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.282420 (+ 312us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.282501 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471}
I20260924 07:17:55.283156 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.266323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:46107 (local address 127.15.127.129:59165)
0924 07:17:55.273283 (+ 6960us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:55.273299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:17:55.273937 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:17:55.275531 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:17:55.275534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:17:55.275553 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:17:55.275633 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.275638 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.276714 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.276718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.280927 (+ 4209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:17:55.280938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.282198 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.282202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:17:55.282287 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.282308 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:17:55.282958 (+ 650us) client_negotiation.cc:240] Negotiation successful
0924 07:17:55.283034 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6906,"thread_start_us":94,"threads_started":1}
I20260924 07:17:55.283198 27094 catalog_manager.cc:5495] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ff81e03f50e4af4bf4b01d86252fc6e (127.15.127.129). New cstate: current_term: 1 leader_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43657 } health_report { overall_health: HEALTHY } } }
I20260924 07:17:55.283882 27397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817c88c44a8f4d278c54e51664d6df05" candidate_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6976a9722563476eaffd3601df9f9089"
I20260924 07:17:55.284126 27397 raft_consensus.cc:3037] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:17:55.285560 27397 raft_consensus.cc:2454] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ff81e03f50e4af4bf4b01d86252fc6e in term 1.
I20260924 07:17:55.283840 27399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "817c88c44a8f4d278c54e51664d6df05" candidate_uuid: "8ff81e03f50e4af4bf4b01d86252fc6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6976a9722563476eaffd3601df9f9089" is_pre_election: true
I20260924 07:17:55.285944 27399 raft_consensus.cc:2362] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8ff81e03f50e4af4bf4b01d86252fc6e in term 1. Re-sending same reply.
I20260924 07:17:55.327852 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.322305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.15.127.129:43657)
0924 07:17:55.322685 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 07:17:55.322690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:17:55.322704 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:17:55.322790 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:17:55.322794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:17:55.322863 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:17:55.322987 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:17:55.323330 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.324161 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.325525 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:17:55.325886 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:17:55.325985 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:17:55.326422 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:17:55.327113 (+ 691us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:17:55.327233 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:17:55.327624 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 07:17:55.327691 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260924 07:17:55.339046 27525 raft_consensus.cc:1261] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 1 FOLLOWER]: Refusing update from remote peer 8ff81e03f50e4af4bf4b01d86252fc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:17:55.339967 27610 consensus_queue.cc:1035] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ceabe26c7c341248c91c57e5766cf2c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 37359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:17:55.348013 27399 raft_consensus.cc:1261] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 1 FOLLOWER]: Refusing update from remote peer 8ff81e03f50e4af4bf4b01d86252fc6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:17:55.348734 27610 consensus_queue.cc:1035] T 817c88c44a8f4d278c54e51664d6df05 P 8ff81e03f50e4af4bf4b01d86252fc6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6976a9722563476eaffd3601df9f9089" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 46107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:17:55.355629 27613 mvcc.cc:205] Tried to move back new op lower bound from 7332799591779065856 to 7332799591549497344. Current Snapshot: MvccSnapshot[applied={T|T < 7332799591779065856}]
I20260924 07:17:55.356101 27614 mvcc.cc:205] Tried to move back new op lower bound from 7332799591779065856 to 7332799591549497344. Current Snapshot: MvccSnapshot[applied={T|T < 7332799591779065856}]
I20260924 07:17:55.362905 27616 mvcc.cc:205] Tried to move back new op lower bound from 7332799591779065856 to 7332799591549497344. Current Snapshot: MvccSnapshot[applied={T|T < 7332799591779065856}]
I20260924 07:17:55.374387 27095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:17:55.374599 27095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:17:55.377679 27095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb889eb74b0a48af8658697042e710b9: Sending DeleteTablet for 3 replicas of tablet 817c88c44a8f4d278c54e51664d6df05
I20260924 07:17:55.378679 27233 tablet_service.cc:1514] Processing DeleteTablet for tablet 817c88c44a8f4d278c54e51664d6df05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:17:55 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:53660
I20260924 07:17:55.379184 27497 tablet_service.cc:1514] Processing DeleteTablet for tablet 817c88c44a8f4d278c54e51664d6df05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:17:55 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:59718
I20260924 07:17:55.379660 27360 tablet_service.cc:1514] Processing DeleteTablet for tablet 817c88c44a8f4d278c54e51664d6df05 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:17:55 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:55844
I20260924 07:17:55.380149 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 27176
I20260924 07:17:55.381481 27624 tablet_replica.cc:330] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: stopping tablet replica
I20260924 07:17:55.381794 27624 raft_consensus.cc:2229] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:17:55.382112 27624 raft_consensus.cc:2258] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:17:55.383010 27623 tablet_replica.cc:330] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: stopping tablet replica
I20260924 07:17:55.383328 27623 raft_consensus.cc:2229] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:17:55.384078 27623 raft_consensus.cc:2258] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:17:55.385181 27624 ts_tablet_manager.cc:1905] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:17:55.388517 27624 ts_tablet_manager.cc:1918] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:17:55.388633 27624 log.cc:1196] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/817c88c44a8f4d278c54e51664d6df05
I20260924 07:17:55.388978 27624 ts_tablet_manager.cc:1939] T 817c88c44a8f4d278c54e51664d6df05 P 6976a9722563476eaffd3601df9f9089: Deleting consensus metadata
I20260924 07:17:55.390520 27082 catalog_manager.cc:4841] TS 6976a9722563476eaffd3601df9f9089 (127.15.127.130:46107): tablet 817c88c44a8f4d278c54e51664d6df05 (table test-table [id=8abfc82448124f56a636b83de0208a4c]) successfully deleted
I20260924 07:17:55.392449 27623 ts_tablet_manager.cc:1905] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:17:55.395522 27623 ts_tablet_manager.cc:1918] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:17:55.395706 27623 log.cc:1196] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/817c88c44a8f4d278c54e51664d6df05
I20260924 07:17:55.396139 27623 ts_tablet_manager.cc:1939] T 817c88c44a8f4d278c54e51664d6df05 P 4ceabe26c7c341248c91c57e5766cf2c: Deleting consensus metadata
W20260924 07:17:55.397076 27080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:43657 (error 108)
I20260924 07:17:55.398002 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 27315
I20260924 07:17:55.398190 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:17:55.397853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43657 (local address 127.0.0.1:53670)
0924 07:17:55.398030 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:17:55.398102 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 07:17:55.400070 27080 catalog_manager.cc:4584] TS 8ff81e03f50e4af4bf4b01d86252fc6e (127.15.127.129:43657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 817c88c44a8f4d278c54e51664d6df05: Network error: Client connection negotiation failed: client connection to 127.15.127.129:43657: connect: Connection refused (error 111)
I20260924 07:17:55.400107 27081 catalog_manager.cc:4841] TS 4ceabe26c7c341248c91c57e5766cf2c (127.15.127.131:37359): tablet 817c88c44a8f4d278c54e51664d6df05 (table test-table [id=8abfc82448124f56a636b83de0208a4c]) successfully deleted
I20260924 07:17:55.405932 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 27450
I20260924 07:17:55.412629 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 27063
I20260924 07:17:55.417874 15870 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:17:55.763106 15870 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:17:55.763285 15870 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:17:56.108675 15870 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:17:56.108 UTC [26649] LOG: received smart shutdown request
2026-09-24 07:17:56.111 UTC [26649] LOG: background worker "logical replication launcher" (PID 26658) exited with exit code 1
2026-09-24 07:17:56.112 UTC [26653] LOG: shutting down
2026-09-24 07:17:56.271 UTC [26649] LOG: database system is shut down
2026-09-24T07:17:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45721 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:17:56 dist-test-slave-zx46 krb5kdc[27634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:17:56 dist-test-slave-zx46 krb5kdc[27634](info): set up 2 sockets
Sep 24 07:17:56 dist-test-slave-zx46 krb5kdc[27634](info): commencing operation
krb5kdc: starting...
W20260924 07:17:57.449141 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:17:57 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:17:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:17:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/rangeradmin_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/rangeradmin_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/rangerlookup_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/rangerlookup_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
I20260924 07:17:57.609913 15870 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPYFHG9/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/postgres -l logfile start
2026-09-24 07:18:00.095 UTC [27664] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:18:00.095 UTC [27664] LOG: listening on IPv4 address "127.15.127.148", port 38371
2026-09-24 07:18:00.099 UTC [27664] LOG: listening on Unix socket "/tmp/.s.PGSQL.38371"
2026-09-24 07:18:00.134 UTC [27667] LOG: database system was shut down at 2026-09-24 07:17:58 UTC
2026-09-24 07:18:00.138 UTC [27664] LOG: database system is ready to accept connections
I20260924 07:18:01.084924 15870 mini_postgres.cc:96] Postgres bound to 38371
2026-09-24 07:18:01.091 UTC [27675] FATAL: database "slave" does not exist
127.15.127.148:38371 - accepting connections
I20260924 07:18:01.091821 15870 mini_ranger.cc:161] Starting Ranger...
I20260924 07:18:01.106397 15870 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:18:01.567735 15870 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:18:01.567806 15870 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-admin
2026-09-24 07:18:01,843 [I] DB FLAVOR :POSTGRES
2026-09-24 07:18:01,844 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:18:01,844 [I] Checking connection..
2026-09-24 07:18:01,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:18:02,096 [I] Checking connection passed.
2026-09-24 07:18:02,096 [I] --------- Verifying version history table ---------
2026-09-24 07:18:02,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:18:02,350 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:18:02,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:18:02,582 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:18:02,582 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:18:02,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:18:02.802 UTC [27739] WARNING: there is no transaction in progress
2026-09-24 07:18:02,809 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:18:02,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:18:03,052 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:18:03,052 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:18:03,052 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:18:03,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:18:03,531 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zx46.c.gcp-upstream.internal','N') ;"
2026-09-24 07:18:03,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:18:04,001 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:18:04,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:18:04,264 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:18:04,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:18:04,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:18:04,758 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:18:04,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:18:05.709 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.726 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.743 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.758 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.773 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.791 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.846 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.853 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.867 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.878 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.888 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.901 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.911 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.917 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.924 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:05.933 UTC [27847] WARNING: there is no transaction in progress
2026-09-24 07:18:06,577 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:18:06,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:18:06,810 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:18:06,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:18:07,088 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:18:07,088 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:18:07,345 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:18:07,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:18:07,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:18:07,846 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zx46.c.gcp-upstream.internal';"
2026-09-24 07:18:08,089 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:18:08,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:18:08,319 [I] DB_PATCHES have already been applied
I20260924 07:18:08.325207 15870 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPYFHG9/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:18:08.325268 15870 mini_ranger.cc:193] Using host: 127.15.127.148
I20260924 07:18:08.328738 15870 mini_ranger.cc:227] Ranger admin URL: http://127.15.127.148:54335
Sep 24, 2026 7:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:18:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:18:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:18:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:18:09.440786 15870 mini_ranger.cc:160] Time spent starting Ranger: real 8.349s user 0.001s sys 0.008s
Sep 24, 2026 7:18:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:18:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:18:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:18:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:18:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:18:31.125108 15870 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.127.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.127.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/rangerkms_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.127.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/rangerkms_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.127.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.127.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.127.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/HTTP_127.15.127.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:18:31 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234311, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:18:31.209229 15870 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:18:31.218263 15870 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:18:32.087769 15870 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:18:32.087858 15870 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms
I20260924 07:18:32.087886 15870 mini_ranger_kms.cc:227] Using postgres at 127.15.127.148:38371
2026-09-24 07:18:32,290 [I] DB FLAVOR :POSTGRES
2026-09-24 07:18:32,290 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:18:32,290 [I] Checking connection
2026-09-24 07:18:32,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:18:32,526 [I] connection success
2026-09-24 07:18:32,526 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:18:32,526 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:18:32,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:18:32,755 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:18:32,755 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:18:32,755 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:18:32,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.127.148:38371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:18:33,016 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:18:33.455204 15870 mini_ranger_kms.cc:310] Created kms service
I20260924 07:18:33.654249 15870 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:18:33.697252 15870 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:18:33 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234311, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:34.114081 15870 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:18:34.114238 15870 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPYFHG9/build/debug/bin/postgresql.jar:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPYFHG9/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:18:34.114260 15870 mini_ranger_kms.cc:240] Using host: 127.15.127.148
I20260924 07:18:34.118064 15870 mini_ranger_kms.cc:276] Ranger KMS PID: 28027
I20260924 07:18:34.118155 15870 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.127.148:54723
Sep 24, 2026 7:18:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:18:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:18:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:18:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:18:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:18:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 07:18:36 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234316, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.127.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:36.329370 15870 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.120s user 0.016s sys 0.170s
I20260924 07:18:36.329491 15870 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:18:36.329561 15870 mini_ranger_kms.cc:401] 127.15.127.148:54723/kms/v1/keys
Sep 24 07:18:36 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234316, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.127.148@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
Sep 24 07:18:37 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234316, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.127.148@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
Sep 24 07:18:39 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:18:39.268430 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:43135
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:43135
--ranger_config_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:39.388315 28078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:39.388581 28078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:39.388619 28078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:39.392658 28078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:39.392712 28078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:39.392732 28078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:39.392748 28078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:39.392763 28078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:39.398241 28078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.127.148:54723/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:43135
--ranger_config_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:39.399240 28078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:39.400259 28078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:18:39.408777 28078 server_base.cc:1034] running on GCE node
W20260924 07:18:39.408818 28083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:39.408959 28086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:39.409117 28084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:39.409720 28078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:39.410813 28078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:39.412011 28078 hybrid_clock.cc:648] HybridClock initialized: now 1790234319411989 us; error 42 us; skew 500 ppm
Sep 24 07:18:39 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:39.415112 28078 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:39.416189 28078 webserver.cc:460] Webserver started at http://127.15.127.190:40381/ using document root <none> and password file <none>
I20260924 07:18:39.416851 28078 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:39.416932 28078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:39.417188 28078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:18:39 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:39.462978 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "8593d79526d146faa0f95a66b5ebbc05"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-zx46"
server_key: "b25abe6b620290829bd523569ab363d1"
server_key_iv: "e0bc1d25df7a03bfe8bbb327579acc66"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:39.463603 28078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "8593d79526d146faa0f95a66b5ebbc05"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-zx46"
server_key: "b25abe6b620290829bd523569ab363d1"
server_key_iv: "e0bc1d25df7a03bfe8bbb327579acc66"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:39.467763 28078 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 07:18:39.482146 28093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:39.483619 28078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:18:39.483767 28078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "8593d79526d146faa0f95a66b5ebbc05"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-zx46"
server_key: "b25abe6b620290829bd523569ab363d1"
server_key_iv: "e0bc1d25df7a03bfe8bbb327579acc66"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:39.483917 28078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:39.503458 28078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:39.506477 28078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:39.506671 28078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:39.514665 28078 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:43135
I20260924 07:18:39.514672 28145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:43135 every 8 connection(s)
I20260924 07:18:39.515698 28078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:39.516544 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28078
I20260924 07:18:39.516674 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:18:39.516952 15870 external_mini_cluster.cc:1428] Setting key 987094414828baa8b1ff097cb09949fb
I20260924 07:18:39.520447 28146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 07:18:39 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:39.527932 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Bootstrap starting.
I20260924 07:18:39.530619 28146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:39.531447 28149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:39.518479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.15.127.190:43135)
0924 07:18:39.518982 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:39.518996 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:39.519030 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:18:39.519687 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:39.519704 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:39.520024 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:39.520356 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:39.521228 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.522200 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.523083 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:39.523624 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:39.523814 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:39.526698 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:39.526703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:39.526716 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:39.526745 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:39.528602 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.529062 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.529070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.529076 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.529188 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:39.529502 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:39.529505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:39.529507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:39.529914 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:39.530112 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:39.530411 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 07:18:39.530705 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":179,"threads_started":1}
I20260924 07:18:39.532862 28146 log.cc:824] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:39.535276 28146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: No bootstrap required, opened a new log
I20260924 07:18:39.539124 28146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:39.539583 28146 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:39.539690 28146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8593d79526d146faa0f95a66b5ebbc05, State: Initialized, Role: FOLLOWER
I20260924 07:18:39.540246 28146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:39.540412 28146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:39.540467 28146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:39.540563 28146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:39.541772 28146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:39.542147 28146 leader_election.cc:302] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8593d79526d146faa0f95a66b5ebbc05; no voters:
I20260924 07:18:39.542501 28146 leader_election.cc:288] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:39.542620 28151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:39.542852 28151 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 LEADER]: Becoming Leader. State: Replica: 8593d79526d146faa0f95a66b5ebbc05, State: Running, Role: LEADER
I20260924 07:18:39.543247 28151 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:39.543831 28146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:39.544436 28151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8593d79526d146faa0f95a66b5ebbc05. Latest consensus state: current_term: 1 leader_uuid: "8593d79526d146faa0f95a66b5ebbc05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } } }
I20260924 07:18:39.544556 28151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:39.544961 28153 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8593d79526d146faa0f95a66b5ebbc05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } } }
I20260924 07:18:39.545198 28153 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:39.546079 28146 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:18:39.546537 28157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:39.554019 28157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:39.560822 28157 catalog_manager.cc:1347] Generated new cluster ID: 8ec8d45d5ce94a80b638324f058d5202
I20260924 07:18:39.560889 28157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:39.568086 28157 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:39.569408 28157 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:39.582890 28157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Generated new TSK 0
I20260924 07:18:39.583758 28157 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:18:40.268067 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43135
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:40.404796 28192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:40.405217 28192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:40.405334 28192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:40.410516 28192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:40.410637 28192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:40.410777 28192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:18:40.416546 28192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.127.148:54723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:40.417734 28192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:40.418744 28192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:40.428244 28199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.428287 28202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.428367 28200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:40.428735 28192 server_base.cc:1034] running on GCE node
I20260924 07:18:40.429214 28192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:40.430025 28192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:40.431242 28192 hybrid_clock.cc:648] HybridClock initialized: now 1790234320431189 us; error 59 us; skew 500 ppm
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:40.434913 28192 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:18:40.436007 28192 webserver.cc:460] Webserver started at http://127.15.127.129:43471/ using document root <none> and password file <none>
I20260924 07:18:40.436726 28192 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:40.436826 28192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:40.437108 28192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:40.459177 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "c232ca049848545d63446f45dab8cd08"
server_key_iv: "7e8bf846bc8d0b636a2ffdc6db78c220"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:40.459794 28192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "c232ca049848545d63446f45dab8cd08"
server_key_iv: "7e8bf846bc8d0b636a2ffdc6db78c220"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:40.463726 28192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 07:18:40.481513 28209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.483374 28192 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 07:18:40.483590 28192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "c232ca049848545d63446f45dab8cd08"
server_key_iv: "7e8bf846bc8d0b636a2ffdc6db78c220"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:40.483846 28192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:40.504228 28192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:40.507962 28192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:40.508189 28192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:40.508879 28192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:40.509876 28192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:40.509964 28192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.510040 28192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:40.510080 28192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.519840 28192 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:35541
I20260924 07:18:40.519903 28323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:35541 every 8 connection(s)
I20260924 07:18:40.520917 28192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:40.525978 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28192
I20260924 07:18:40.526127 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:40.526482 15870 external_mini_cluster.cc:1428] Setting key e818e02eb2627e77496e456ff092e722
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:18:40.539280 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.523393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:43793 (local address 127.15.127.190:43135)
0924 07:18:40.525748 (+ 2355us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:40.525758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:40.525778 (+ 20us) server_negotiation.cc:407] Connection header received
0924 07:18:40.525975 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:40.525979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:40.526037 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:40.526187 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:40.527785 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.528605 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.530153 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.530568 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.530655 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.534176 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:40.534183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:40.534190 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:40.534225 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:40.536316 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.537091 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.537095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.537098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.537214 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.537714 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.537719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.537721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.537996 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:40.538107 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:40.538888 (+ 781us) server_negotiation.cc:299] Negotiation successful
0924 07:18:40.539043 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2263,"thread_start_us":101,"threads_started":1}
I20260924 07:18:40.541356 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.523896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43135 (local address 127.15.127.129:43793)
0924 07:18:40.524525 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:40.524583 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:40.525711 (+ 1128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:40.526620 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:40.526629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:40.526974 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:40.527610 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:40.527629 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.528785 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.528789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:40.529985 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:40.530001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.530881 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.530887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:40.531135 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.531861 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:40.531889 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:40.533977 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:40.536478 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:40.536488 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:40.536509 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:40.536903 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:40.537365 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:40.537368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:40.537371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:40.537560 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:40.538193 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:40.538203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:40.538566 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 07:18:40.539793 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0924 07:18:40.540167 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":152,"threads_started":1}
I20260924 07:18:40.543006 28324 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43135
I20260924 07:18:40.543264 28324 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:40.543900 28324 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:40.545944 28110 ts_manager.cc:194] Registered new tserver with Master: d2cc4ab5f7dc4f29a1e4a619c153f651 (127.15.127.129:35541)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:18:40.547533 28110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:43793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:18:40.571806 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43135
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:18:40.726861 28331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:40.727348 28331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:40.727527 28331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:40.733873 28331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:40.733965 28331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:40.734054 28331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:18:40.742208 28331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.127.148:54723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:40.743649 28331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:40.745043 28331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:40.757797 28340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.758112 28338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:40.758777 28337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:40.759205 28331 server_base.cc:1034] running on GCE node
I20260924 07:18:40.759697 28331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:40.760454 28331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:40.761636 28331 hybrid_clock.cc:648] HybridClock initialized: now 1790234320761612 us; error 37 us; skew 500 ppm
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:40.765291 28331 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:18:40.766407 28331 webserver.cc:460] Webserver started at http://127.15.127.130:40257/ using document root <none> and password file <none>
I20260924 07:18:40.767170 28331 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:40.767266 28331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:40.767568 28331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:40.791705 28331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "e7a38f736b90821899ec4cc4894a4ec9"
server_key_iv: "6c7e61f4852b033fb739325d4f48b2ea"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:40.792416 28331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "e7a38f736b90821899ec4cc4894a4ec9"
server_key_iv: "6c7e61f4852b033fb739325d4f48b2ea"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:40.797614 28331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 07:18:40.823889 28347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.825639 28331 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260924 07:18:40.825747 28331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "e7a38f736b90821899ec4cc4894a4ec9"
server_key_iv: "6c7e61f4852b033fb739325d4f48b2ea"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:40.825914 28331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:40.852707 28331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:40.856504 28331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:40.856766 28331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:40.857538 28331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:40.858618 28331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:40.858728 28331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.858834 28331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:40.858920 28331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:40.875192 28331 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:42419
I20260924 07:18:40.877982 28331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:40.879587 28460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:42419 every 8 connection(s)
I20260924 07:18:40.886469 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28331
I20260924 07:18:40.886586 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:40.886870 15870 external_mini_cluster.cc:1428] Setting key cd89a55941baa832b3c666eea36064e3
Sep 24 07:18:40 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:40.896309 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.882114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43135 (local address 127.15.127.130:45317)
0924 07:18:40.882596 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:40.882649 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:40.883752 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:40.884304 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:40.884310 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:40.884578 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:40.885127 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:40.885143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.886190 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.886193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:40.887594 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:40.887606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.888415 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.888418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:40.888609 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.889296 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:40.889317 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:40.891014 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:40.893150 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:40.893156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:40.893170 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:40.893512 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:40.893838 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:40.893841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:40.893843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:40.894013 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:40.894498 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:40.894505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:40.894805 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 07:18:40.895013 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 07:18:40.895319 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":152,"threads_started":1}
I20260924 07:18:40.897140 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:40.881651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:45317 (local address 127.15.127.190:43135)
0924 07:18:40.881806 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:40.881811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:40.882748 (+ 937us) server_negotiation.cc:407] Connection header received
0924 07:18:40.883994 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:40.883998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:40.884056 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:40.884134 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:40.885267 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.886030 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.887739 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:40.888102 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:40.888184 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:40.891210 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:40.891214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:40.891217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:40.891244 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:40.893023 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.893635 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.893638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.893640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.893736 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:40.894123 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:40.894127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:40.894129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:40.894339 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:40.894445 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:40.896838 (+ 2393us) server_negotiation.cc:299] Negotiation successful
0924 07:18:40.896967 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:18:40.898219 28461 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43135
I20260924 07:18:40.898463 28461 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:40.898999 28461 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:40.900072 28110 ts_manager.cc:194] Registered new tserver with Master: 7b24d3e0bf4c4fb5ba47d222b9ed61af (127.15.127.130:42419)
I20260924 07:18:40.900806 28110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:45317
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:18:40.922508 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43135
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234320, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
W20260924 07:18:41.057775 28466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:41.058144 28466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:41.058254 28466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:41.064101 28466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:41.064179 28466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:41.064278 28466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:18:41.071821 28466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.127.148:54723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:43135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:41.073261 28466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:41.074587 28466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:41.087242 28472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.087625 28474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.088245 28471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.089052 28466 server_base.cc:1034] running on GCE node
I20260924 07:18:41.089535 28466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:41.090323 28466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:41.093585 28466 hybrid_clock.cc:648] HybridClock initialized: now 1790234321093482 us; error 109 us; skew 500 ppm
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:41.097275 28466 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:18:41.098357 28466 webserver.cc:460] Webserver started at http://127.15.127.131:36993/ using document root <none> and password file <none>
I20260924 07:18:41.099067 28466 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:41.099150 28466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:41.099395 28466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:41.114800 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "132387b04d7e438ab75324b60dbeb8fa"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-zx46"
server_key: "a66b6beca27ab03441c7ae7788829550"
server_key_iv: "e06e8f256a26b0e2f3833ea232e3eecf"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:41.115561 28466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "132387b04d7e438ab75324b60dbeb8fa"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-zx46"
server_key: "a66b6beca27ab03441c7ae7788829550"
server_key_iv: "e06e8f256a26b0e2f3833ea232e3eecf"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:41.131094 28466 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.006s sys 0.000s
I20260924 07:18:41.147181 28481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.148936 28466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:41.149115 28466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "132387b04d7e438ab75324b60dbeb8fa"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-zx46"
server_key: "a66b6beca27ab03441c7ae7788829550"
server_key_iv: "e06e8f256a26b0e2f3833ea232e3eecf"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:41.149271 28466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:41.186384 28466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:41.190234 28466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:41.190435 28466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:41.191126 28466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:41.192185 28466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:41.192256 28466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.192314 28466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:41.192345 28466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.205858 28466 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:39215
I20260924 07:18:41.207031 28466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:41.213908 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28466
I20260924 07:18:41.214051 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:18:41.214355 15870 external_mini_cluster.cc:1428] Setting key 8c4141c688509a1e6bed845da2a8bf7a
I20260924 07:18:41.208350 28594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:39215 every 8 connection(s)
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:41.225970 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.211637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:41017 (local address 127.15.127.190:43135)
0924 07:18:41.211773 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:41.211778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:41.212574 (+ 796us) server_negotiation.cc:407] Connection header received
0924 07:18:41.213864 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:41.213869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:41.213930 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:41.213991 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:41.215602 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.216337 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.218091 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.218467 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.218552 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.221748 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:41.221752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:41.221755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:41.221783 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:41.223584 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.224189 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.224192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.224194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.224275 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.224681 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.224684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.224686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.224897 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:41.224987 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:41.225673 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 07:18:41.225802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:18:41.227813 28597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.210547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43135 (local address 127.15.127.131:41017)
0924 07:18:41.212399 (+ 1852us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:41.212465 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:41.213605 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:41.214408 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:41.214418 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:41.214771 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:41.215432 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.215463 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.216495 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.216498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.217953 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.217965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.218764 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.218769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.218981 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.219737 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:41.219757 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:41.221559 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:41.223724 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.223729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.223745 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.224071 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.224381 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.224384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.224386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.224570 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.225071 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:41.225077 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:41.225372 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 07:18:41.226402 (+ 1030us) client_negotiation.cc:240] Negotiation successful
0924 07:18:41.226737 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1639,"thread_start_us":121,"threads_started":1}
I20260924 07:18:41.229297 28595 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43135
I20260924 07:18:41.229557 28595 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:41.230103 28595 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:41.231246 28110 ts_manager.cc:194] Registered new tserver with Master: 132387b04d7e438ab75324b60dbeb8fa (127.15.127.131:39215)
I20260924 07:18:41.231999 28110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:41017
I20260924 07:18:41.239575 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:41.240425 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28192
I20260924 07:18:41.248850 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28331
I20260924 07:18:41.257761 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28466
I20260924 07:18:41.265951 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28078
I20260924 07:18:41.272715 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:43135
--webserver_interface=127.15.127.190
--webserver_port=40381
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:43135
--ranger_config_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:41.380543 28598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:41.380808 28598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:41.380848 28598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:41.385104 28598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:41.385164 28598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:41.385183 28598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:41.385200 28598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:41.385247 28598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:41.390769 28598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:43135
--ranger_config_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:43135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=40381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:41.391834 28598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:41.392930 28598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:41.402424 28603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.402608 28598 server_base.cc:1034] running on GCE node
W20260924 07:18:41.402405 28604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.402396 28606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.403400 28598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:41.404505 28598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:41.405689 28598 hybrid_clock.cc:648] HybridClock initialized: now 1790234321405676 us; error 30 us; skew 500 ppm
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:41.408421 28598 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:41.409430 28598 webserver.cc:460] Webserver started at http://127.15.127.190:40381/ using document root <none> and password file <none>
I20260924 07:18:41.410003 28598 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:41.410070 28598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:41.465662 28598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:41.467993 28614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.469512 28598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:41.469657 28598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "8593d79526d146faa0f95a66b5ebbc05"
format_stamp: "Formatted at 2026-09-24 07:18:39 on dist-test-slave-zx46"
server_key: "b25abe6b620290829bd523569ab363d1"
server_key_iv: "e0bc1d25df7a03bfe8bbb327579acc66"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:41.470242 28598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:41.483067 28598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:41.486202 28598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:41.486402 28598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:41.494462 28598 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:43135
I20260924 07:18:41.494464 28666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:43135 every 8 connection(s)
I20260924 07:18:41.495549 28598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:41.499989 28667 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:18:41.500082 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28598
I20260924 07:18:41.500588 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:35541
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:43135
--webserver_port=43471
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:41.502709 28667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Bootstrap starting.
I20260924 07:18:41.524480 28667 log.cc:824] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:41.533537 28667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:41.534137 28667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Bootstrap complete.
I20260924 07:18:41.540122 28667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:41.540771 28667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8593d79526d146faa0f95a66b5ebbc05, State: Initialized, Role: FOLLOWER
I20260924 07:18:41.541544 28667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:41.541815 28667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:41.541939 28667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:41.542083 28667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:18:41.544656 28667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:41.545127 28667 leader_election.cc:302] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8593d79526d146faa0f95a66b5ebbc05; no voters:
I20260924 07:18:41.545552 28667 leader_election.cc:288] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:18:41.546939 28667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:41.549301 28667 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:18:41.550518 28671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:18:41.550800 28671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [term 2 LEADER]: Becoming Leader. State: Replica: 8593d79526d146faa0f95a66b5ebbc05, State: Running, Role: LEADER
I20260924 07:18:41.551177 28671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } }
I20260924 07:18:41.552222 28671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8593d79526d146faa0f95a66b5ebbc05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } } }
I20260924 07:18:41.552423 28671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:41.552651 28671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8593d79526d146faa0f95a66b5ebbc05. Latest consensus state: current_term: 2 leader_uuid: "8593d79526d146faa0f95a66b5ebbc05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8593d79526d146faa0f95a66b5ebbc05" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 43135 } } }
I20260924 07:18:41.552783 28671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:41.554972 28686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:41.558338 28686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:41.559000 28686 catalog_manager.cc:1259] Loaded cluster ID: 8ec8d45d5ce94a80b638324f058d5202
I20260924 07:18:41.559113 28686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:41.561582 28686 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:41.564983 28686 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Loaded TSK: 0
I20260924 07:18:41.566881 28686 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:18:41.663247 28669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:41.663606 28669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:41.663682 28669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:41.670014 28669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:41.670115 28669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:41.670219 28669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:18:41.678256 28669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:35541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=43471
--tserver_master_addrs=127.15.127.190:43135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:41.679819 28669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:41.681272 28669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:41.694435 28699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:41.694941 28702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.696131 28669 server_base.cc:1034] running on GCE node
W20260924 07:18:41.694435 28700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:41.697636 28669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:41.698462 28669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:41.713954 28669 hybrid_clock.cc:648] HybridClock initialized: now 1790234321713891 us; error 78 us; skew 500 ppm
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:41.718169 28669 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:18:41.719444 28669 webserver.cc:460] Webserver started at http://127.15.127.129:43471/ using document root <none> and password file <none>
I20260924 07:18:41.720185 28669 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:41.720285 28669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:41.748781 28669 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260924 07:18:41.762581 28715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.765843 28669 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.000s
I20260924 07:18:41.765991 28669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "c232ca049848545d63446f45dab8cd08"
server_key_iv: "7e8bf846bc8d0b636a2ffdc6db78c220"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:41.766752 28669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:41.824394 28669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:41.832012 28669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:41.832204 28669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:41.832811 28669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:41.833824 28669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:41.833894 28669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.833950 28669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:41.833976 28669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:41.861402 28828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:35541 every 8 connection(s)
I20260924 07:18:41.869740 28669 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:35541
I20260924 07:18:41.871336 28669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:41.879458 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28669
I20260924 07:18:41.880007 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:42419
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:43135
--webserver_port=40257
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:41 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234321, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:41.911885 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.877903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60779 (local address 127.15.127.190:43135)
0924 07:18:41.885266 (+ 7363us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:41.885289 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:41.885337 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:18:41.885643 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:41.885659 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:41.886099 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:41.886497 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:41.889612 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.891242 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.892966 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.893867 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.894039 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.897807 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:41.897813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:41.897829 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:41.897858 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:41.899864 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.903202 (+ 3338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.903208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.903213 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.903357 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:41.904431 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:41.904435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:41.904437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:41.905794 (+ 1357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:41.905922 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:41.906865 (+ 943us) server_negotiation.cc:299] Negotiation successful
0924 07:18:41.907232 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7179,"thread_start_us":114,"threads_started":1}
I20260924 07:18:41.913877 28831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:41.874710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43135 (local address 127.15.127.129:60779)
0924 07:18:41.878648 (+ 3938us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:41.878717 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:41.882237 (+ 3520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:41.888322 (+ 6085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:41.888334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:41.888703 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:41.889401 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.889420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.891398 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.891402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.892829 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:41.892840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:41.894256 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:41.894260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:41.894445 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:41.895143 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:41.895166 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:41.897596 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:41.900416 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.900423 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.900437 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.903062 (+ 2625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.903472 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:41.903475 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:41.903477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:41.904050 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:41.905992 (+ 1942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:41.905998 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:41.906312 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 07:18:41.906513 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:18:41.912515 (+ 6002us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3653,"thread_start_us":134,"threads_started":1}
I20260924 07:18:41.916637 28829 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43135
I20260924 07:18:41.916874 28829 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:41.917443 28829 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
W20260924 07:18:41.919456 28829 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:43135 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:18:41.919569 28829 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:41.919723 28829 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:41.920418 28829 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:41.920572 28829 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
W20260924 07:18:41.921268 28829 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:18:42.066952 28833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:42.067896 28833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:42.068066 28833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:42.075480 28833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:42.075574 28833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:42.075670 28833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:18:42.083515 28833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:42419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=40257
--tserver_master_addrs=127.15.127.190:43135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:42.084985 28833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:42.086531 28833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:42.100198 28839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.101121 28841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.101418 28833 server_base.cc:1034] running on GCE node
W20260924 07:18:42.101671 28838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.102149 28833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:42.102952 28833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:42.104107 28833 hybrid_clock.cc:648] HybridClock initialized: now 1790234322104072 us; error 49 us; skew 500 ppm
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:42.113759 28833 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:18:42.115077 28833 webserver.cc:460] Webserver started at http://127.15.127.130:40257/ using document root <none> and password file <none>
I20260924 07:18:42.115835 28833 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:42.115988 28833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:42.140949 28833 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:18:42.143859 28849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.145365 28833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:42.145519 28833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af"
format_stamp: "Formatted at 2026-09-24 07:18:40 on dist-test-slave-zx46"
server_key: "e7a38f736b90821899ec4cc4894a4ec9"
server_key_iv: "6c7e61f4852b033fb739325d4f48b2ea"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:42.146222 28833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:42.172628 28833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:42.176182 28833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:42.176383 28833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:42.177361 28833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:42.178278 28833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:42.178346 28833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.178406 28833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:42.178439 28833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.189200 28833 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:42419
I20260924 07:18:42.189761 28962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:42419 every 8 connection(s)
I20260924 07:18:42.190286 28833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:42.196270 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28833
I20260924 07:18:42.196769 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.127.148:54723/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:39215
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:43135
--webserver_port=36993
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:42.210987 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.192435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:48607 (local address 127.15.127.190:43135)
0924 07:18:42.192566 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:42.192569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:42.193844 (+ 1275us) server_negotiation.cc:407] Connection header received
0924 07:18:42.195166 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:42.195170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:42.195234 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:42.195294 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:42.199441 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.200257 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.201682 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.202068 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.202178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.204988 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:42.204993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:42.204995 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:42.205047 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:42.206756 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.209132 (+ 2376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.209135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.209137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.209223 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.209644 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.209660 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.209661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.209865 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:42.209973 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:42.210666 (+ 693us) server_negotiation.cc:299] Negotiation successful
0924 07:18:42.210817 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:18:42.212522 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.192912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43135 (local address 127.15.127.130:48607)
0924 07:18:42.193675 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:42.193730 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:42.194893 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:42.195712 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:42.195720 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:42.198540 (+ 2820us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:42.199248 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.199269 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.200431 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.200434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.201550 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.201560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.202353 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.202358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.202553 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.203232 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:42.203253 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:42.204765 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:42.207025 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.207031 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.207046 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.207378 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.209330 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.209333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.209335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.209531 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.210039 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:42.210045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:42.210376 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 07:18:42.211182 (+ 806us) client_negotiation.cc:240] Negotiation successful
0924 07:18:42.211517 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":133,"threads_started":1}
I20260924 07:18:42.213864 28963 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43135
I20260924 07:18:42.214120 28963 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.214656 28963 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
W20260924 07:18:42.216302 28963 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:43135 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:18:42.216428 28963 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.216574 28963 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:42.217345 28963 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.217525 28963 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
W20260924 07:18:42.218254 28963 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:18:42.359624 28966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:42.359990 28966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:42.360059 28966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:42.366236 28966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:42.366320 28966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:42.366406 28966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:18:42.374332 28966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.127.148:54723/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:39215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=36993
--tserver_master_addrs=127.15.127.190:43135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:42.375669 28966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:42.376984 28966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:42.389078 28972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.389097 28973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:42.391641 28975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:42.393394 28966 server_base.cc:1034] running on GCE node
I20260924 07:18:42.393870 28966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:42.394551 28966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:42.395802 28966 hybrid_clock.cc:648] HybridClock initialized: now 1790234322395767 us; error 43 us; skew 500 ppm
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:42.401278 28966 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:18:42.402186 28966 webserver.cc:460] Webserver started at http://127.15.127.131:36993/ using document root <none> and password file <none>
I20260924 07:18:42.402763 28966 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:42.402833 28966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:42.419782 28966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:42.423033 28983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.424669 28966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:18:42.424827 28966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "132387b04d7e438ab75324b60dbeb8fa"
format_stamp: "Formatted at 2026-09-24 07:18:41 on dist-test-slave-zx46"
server_key: "a66b6beca27ab03441c7ae7788829550"
server_key_iv: "e06e8f256a26b0e2f3833ea232e3eecf"
server_key_version: "kuduclusterkey@0"
I20260924 07:18:42.425493 28966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:42.452771 28966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:42.456430 28966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:42.456637 28966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:42.457307 28966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:42.458252 28966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:42.458300 28966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.458374 28966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:42.458405 28966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:42.468923 28966 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:39215
I20260924 07:18:42.468945 29096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:39215 every 8 connection(s)
I20260924 07:18:42.469946 28966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:42.477229 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 28966
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:42.489156 28832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.472032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:33917 (local address 127.15.127.190:43135)
0924 07:18:42.472179 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:42.472185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:42.473299 (+ 1114us) server_negotiation.cc:407] Connection header received
0924 07:18:42.474676 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:42.474680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:42.474736 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:42.474828 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:42.476363 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.477073 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.481995 (+ 4922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.482344 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.482416 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.485224 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:42.485227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:42.485230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:42.485276 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:42.486989 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.487480 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.487483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.487485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.487565 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.487915 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.487918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.487920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.488093 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:42.488217 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:42.488838 (+ 621us) server_negotiation.cc:299] Negotiation successful
0924 07:18:42.488948 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:18:42.489377 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.477915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.15.127.190:43135)
0924 07:18:42.478568 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:42.478573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:42.478585 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:42.478619 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:42.478622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:42.478666 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:42.478758 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:42.479713 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.480495 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.482731 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.483121 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.483255 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.484861 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:42.484865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:42.484867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:42.484892 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:42.486950 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.487667 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.487670 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.487671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.487735 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:42.488380 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:42.488383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:42.488384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:42.488570 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:42.488783 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:42.489109 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 07:18:42.489213 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":117,"threads_started":1}
I20260924 07:18:42.490893 29099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:42.472535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:43135 (local address 127.15.127.131:33917)
0924 07:18:42.473135 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:42.473195 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:42.474427 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:42.475111 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:42.475121 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:42.475488 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:42.476173 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.476195 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.477182 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.477186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.481849 (+ 4663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:42.481860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:42.482631 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:42.482635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:42.482868 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:42.483516 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:42.483534 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:42.485012 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:42.487099 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.487105 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.487122 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.487369 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.487660 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:42.487663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:42.487664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:42.487843 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:42.488217 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:42.488225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:42.488568 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 07:18:42.489850 (+ 1282us) client_negotiation.cc:240] Negotiation successful
0924 07:18:42.490141 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":147,"threads_started":1}
I20260924 07:18:42.492172 29097 heartbeater.cc:347] Connected to a master server at 127.15.127.190:43135
I20260924 07:18:42.492411 29097 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.492911 29097 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
W20260924 07:18:42.494457 29097 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:43135 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:18:42.494567 29097 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.494822 29097 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:42.495610 29097 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.495765 29097 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
W20260924 07:18:42.496518 29097 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 07:18:42 dist-test-slave-zx46 krb5kdc[27634](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:42.921517 28829 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:42.921818 28829 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:42.923008 28622 ts_manager.cc:194] Registered new tserver with Master: d2cc4ab5f7dc4f29a1e4a619c153f651 (127.15.127.129:35541)
I20260924 07:18:42.924686 28622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:60779
I20260924 07:18:43.218459 28963 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:43.218781 28963 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:43.219554 28622 ts_manager.cc:194] Registered new tserver with Master: 7b24d3e0bf4c4fb5ba47d222b9ed61af (127.15.127.130:42419)
I20260924 07:18:43.220314 28622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:48607
Sep 24 07:18:43 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:43.496724 29097 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:43.497013 29097 heartbeater.cc:511] Master 127.15.127.190:43135 requested a full tablet report, sending...
I20260924 07:18:43.501443 28622 ts_manager.cc:194] Registered new tserver with Master: 132387b04d7e438ab75324b60dbeb8fa (127.15.127.131:39215)
I20260924 07:18:43.502197 28622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:33917
I20260924 07:18:43.503870 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:43.517875 29119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.505556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.15.127.190:43135)
0924 07:18:43.508769 (+ 3213us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.508776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.509431 (+ 655us) server_negotiation.cc:407] Connection header received
0924 07:18:43.509619 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.509621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.509662 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.509753 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:43.510433 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.511003 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.512124 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.512488 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.512590 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.514443 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:43.514447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:43.514449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:43.514474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:43.516055 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:43.516544 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:43.516548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:43.516550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:43.516635 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:43.516952 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:43.516955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:43.516957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:43.517195 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:43.517332 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.517560 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.517679 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3106,"thread_start_us":97,"threads_started":1}
I20260924 07:18:43.521967 28622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:43.526304 28622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 24 07:18:43 dist-test-slave-zx46 krb5kdc[27634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234322, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for HTTP/127.15.127.148@KRBTEST.COM
I20260924 07:18:43.562320 29129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.554093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:39215 (local address 127.0.0.1:36248)
0924 07:18:43.556062 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.556071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.556183 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.556926 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.556931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.556951 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.557122 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.557128 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.558531 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.558533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.559892 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.559900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.561795 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.561799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.561905 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.561942 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.562032 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.562149 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1921,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260924 07:18:43.563162 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.554544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.15.127.131:39215)
0924 07:18:43.555953 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.555958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.556275 (+ 317us) server_negotiation.cc:407] Connection header received
0924 07:18:43.556367 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.556373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.556587 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.556743 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.557240 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.558413 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.560023 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.561638 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.562741 (+ 1103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.562865 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.562925 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.563014 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":71,"threads_started":1}
I20260924 07:18:43.563444 29127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.554654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:35541 (local address 127.0.0.1:41640)
0924 07:18:43.557556 (+ 2902us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.557569 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.557660 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.558278 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.558281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.558296 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.558381 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.558386 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.559722 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.559725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.560780 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.560791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.562041 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.562044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.562133 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.562152 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.562218 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.562274 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2843,"thread_start_us":77,"threads_started":1}
I20260924 07:18:43.565336 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.554988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.15.127.129:35541)
0924 07:18:43.557404 (+ 2416us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.557411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.557761 (+ 350us) server_negotiation.cc:407] Connection header received
0924 07:18:43.557836 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.557841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.558035 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.558161 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.558511 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.559604 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.560920 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.561718 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.561826 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.561979 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.565097 (+ 3118us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.565192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2314,"thread_start_us":104,"threads_started":1}
I20260924 07:18:43.566453 29028 tablet_service.cc:1467] Processing CreateTablet for tablet 3cfde9a7e9a142d887a2d63fa32e0553 (DEFAULT_TABLE table=test-table [id=ded98f93569c4175b7a0f7642b047f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.567413 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.553310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42419 (local address 127.0.0.1:60516)
0924 07:18:43.555477 (+ 2167us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.555534 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.555670 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.563833 (+ 8163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.563836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.563853 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.563951 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.563958 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.565064 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.565067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.566161 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.566173 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.567003 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.567007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.567100 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.567122 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.567181 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.567248 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"thread_start_us":125,"threads_started":1}
I20260924 07:18:43.567677 29028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cfde9a7e9a142d887a2d63fa32e0553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.567704 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.553437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.15.127.130:42419)
0924 07:18:43.562762 (+ 9325us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.562768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.562793 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:18:43.562864 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.562884 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.563077 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.564104 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.564214 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.564915 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.566316 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.566879 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.567088 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.567222 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.567266 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.567367 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9208,"thread_start_us":69,"threads_started":1}
I20260924 07:18:43.570562 28897 tablet_service.cc:1467] Processing CreateTablet for tablet 3cfde9a7e9a142d887a2d63fa32e0553 (DEFAULT_TABLE table=test-table [id=ded98f93569c4175b7a0f7642b047f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.571200 28758 tablet_service.cc:1467] Processing CreateTablet for tablet 3cfde9a7e9a142d887a2d63fa32e0553 (DEFAULT_TABLE table=test-table [id=ded98f93569c4175b7a0f7642b047f26]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:43.571725 28897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cfde9a7e9a142d887a2d63fa32e0553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.572221 28758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cfde9a7e9a142d887a2d63fa32e0553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:43.581202 29131 tablet_bootstrap.cc:492] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Bootstrap starting.
I20260924 07:18:43.583590 29131 tablet_bootstrap.cc:654] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.584453 29131 log.cc:824] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.585210 29132 tablet_bootstrap.cc:492] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: Bootstrap starting.
I20260924 07:18:43.587355 29132 tablet_bootstrap.cc:654] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.588124 29132 log.cc:824] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.589309 29133 tablet_bootstrap.cc:492] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Bootstrap starting.
I20260924 07:18:43.591435 29133 tablet_bootstrap.cc:654] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:43.592170 29133 log.cc:824] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:43.595082 29131 tablet_bootstrap.cc:492] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: No bootstrap required, opened a new log
I20260924 07:18:43.595297 29131 ts_tablet_manager.cc:1397] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260924 07:18:43.599261 29131 raft_consensus.cc:348] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.599582 29131 raft_consensus.cc:374] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.599637 29131 raft_consensus.cc:729] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132387b04d7e438ab75324b60dbeb8fa, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.600122 29131 consensus_queue.cc:260] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.604264 29133 tablet_bootstrap.cc:492] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: No bootstrap required, opened a new log
I20260924 07:18:43.604485 29133 ts_tablet_manager.cc:1397] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260924 07:18:43.605382 29097 heartbeater.cc:503] Master 127.15.127.190:43135 was elected leader, sending a full tablet report...
I20260924 07:18:43.608446 29133 raft_consensus.cc:348] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.608736 29133 raft_consensus.cc:374] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.608794 29133 raft_consensus.cc:729] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b24d3e0bf4c4fb5ba47d222b9ed61af, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.609272 29133 consensus_queue.cc:260] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.610359 29133 ts_tablet_manager.cc:1428] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260924 07:18:43.611150 29131 ts_tablet_manager.cc:1428] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260924 07:18:43.613790 29132 tablet_bootstrap.cc:492] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: No bootstrap required, opened a new log
I20260924 07:18:43.614015 29132 ts_tablet_manager.cc:1397] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20260924 07:18:43.619179 29132 raft_consensus.cc:348] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.619505 29132 raft_consensus.cc:374] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:43.619585 29132 raft_consensus.cc:729] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2cc4ab5f7dc4f29a1e4a619c153f651, State: Initialized, Role: FOLLOWER
I20260924 07:18:43.620077 29132 consensus_queue.cc:260] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.621197 29132 ts_tablet_manager.cc:1428] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260924 07:18:43.621562 28829 heartbeater.cc:503] Master 127.15.127.190:43135 was elected leader, sending a full tablet report...
I20260924 07:18:43.623374 28963 heartbeater.cc:503] Master 127.15.127.190:43135 was elected leader, sending a full tablet report...
W20260924 07:18:43.623904 28830 tablet.cc:2369] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:43.694597 28964 tablet.cc:2369] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:43.721782 29098 tablet.cc:2369] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:43.802861 29137 raft_consensus.cc:482] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:43.803113 29137 raft_consensus.cc:504] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.804646 29137 leader_election.cc:288] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2cc4ab5f7dc4f29a1e4a619c153f651 (127.15.127.129:35541), 7b24d3e0bf4c4fb5ba47d222b9ed61af (127.15.127.130:42419)
I20260924 07:18:43.809226 29126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.805463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:44993 (local address 127.15.127.130:42419)
0924 07:18:43.805591 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.805595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.805606 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:18:43.805651 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.805653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.805688 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.805776 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.806087 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.806753 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.808055 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.808636 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.808699 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.808735 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.809023 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.809091 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:18:43.809238 29149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.805051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42419 (local address 127.15.127.131:44993)
0924 07:18:43.805294 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.805308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.805426 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.805791 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.805793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.805811 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.805977 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.805990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.806857 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.806863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.807927 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.807936 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.808818 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.808820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.808889 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.808904 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.808985 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.809068 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":12,"thread_start_us":95,"threads_started":1}
I20260924 07:18:43.809818 29148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.804939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:35541 (local address 127.15.127.131:51215)
0924 07:18:43.805293 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.805307 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:43.805436 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:43.806175 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:43.806177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:43.806194 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:43.806303 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.806309 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.807355 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.807359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.808417 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:43.808426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.809407 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.809410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:43.809520 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.809542 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:18:43.809620 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:18:43.809684 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20260924 07:18:43.809856 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.805015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:51215 (local address 127.15.127.129:35541)
0924 07:18:43.805146 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.805150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.805388 (+ 238us) server_negotiation.cc:407] Connection header received
0924 07:18:43.806003 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.806007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.806063 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.806156 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:43.806457 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.807237 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.808539 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.809171 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.809293 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.809336 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.809655 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.809738 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:18:43.810093 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfde9a7e9a142d887a2d63fa32e0553" candidate_uuid: "132387b04d7e438ab75324b60dbeb8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" is_pre_election: true
I20260924 07:18:43.810420 28900 raft_consensus.cc:2454] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 132387b04d7e438ab75324b60dbeb8fa in term 0.
I20260924 07:18:43.810540 28770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfde9a7e9a142d887a2d63fa32e0553" candidate_uuid: "132387b04d7e438ab75324b60dbeb8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" is_pre_election: true
I20260924 07:18:43.810806 28770 raft_consensus.cc:2454] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 132387b04d7e438ab75324b60dbeb8fa in term 0.
I20260924 07:18:43.810911 28987 leader_election.cc:302] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132387b04d7e438ab75324b60dbeb8fa, 7b24d3e0bf4c4fb5ba47d222b9ed61af; no voters:
I20260924 07:18:43.811203 29137 raft_consensus.cc:2781] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:43.811290 29137 raft_consensus.cc:482] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:43.811339 29137 raft_consensus.cc:3037] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.812398 29137 raft_consensus.cc:504] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.812740 29137 leader_election.cc:288] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [CANDIDATE]: Term 1 election: Requested vote from peers d2cc4ab5f7dc4f29a1e4a619c153f651 (127.15.127.129:35541), 7b24d3e0bf4c4fb5ba47d222b9ed61af (127.15.127.130:42419)
I20260924 07:18:43.813124 28900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfde9a7e9a142d887a2d63fa32e0553" candidate_uuid: "132387b04d7e438ab75324b60dbeb8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af"
I20260924 07:18:43.813263 28900 raft_consensus.cc:3037] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.814250 28900 raft_consensus.cc:2454] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 132387b04d7e438ab75324b60dbeb8fa in term 1.
I20260924 07:18:43.814522 28770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cfde9a7e9a142d887a2d63fa32e0553" candidate_uuid: "132387b04d7e438ab75324b60dbeb8fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651"
I20260924 07:18:43.814558 28987 leader_election.cc:302] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 132387b04d7e438ab75324b60dbeb8fa, 7b24d3e0bf4c4fb5ba47d222b9ed61af; no voters:
I20260924 07:18:43.814683 28770 raft_consensus.cc:3037] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:43.814754 29137 raft_consensus.cc:2781] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:43.814970 29137 raft_consensus.cc:686] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 1 LEADER]: Becoming Leader. State: Replica: 132387b04d7e438ab75324b60dbeb8fa, State: Running, Role: LEADER
I20260924 07:18:43.815299 29137 consensus_queue.cc:237] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } }
I20260924 07:18:43.815939 28770 raft_consensus.cc:2454] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 132387b04d7e438ab75324b60dbeb8fa in term 1.
I20260924 07:18:43.818497 28624 catalog_manager.cc:5495] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 132387b04d7e438ab75324b60dbeb8fa (127.15.127.131). New cstate: current_term: 1 leader_uuid: "132387b04d7e438ab75324b60dbeb8fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "132387b04d7e438ab75324b60dbeb8fa" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39215 } health_report { overall_health: HEALTHY } } }
I20260924 07:18:43.838317 29128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.834597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.15.127.131:39215)
0924 07:18:43.834741 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:43.834746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:43.834760 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:18:43.834938 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:43.834940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:43.835022 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:43.835134 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:43.835393 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.835910 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.836780 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:43.837046 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:43.837171 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:43.837479 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:43.837893 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:43.838029 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:43.838092 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:18:43.838151 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:18:43.844046 28900 raft_consensus.cc:1261] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 1 FOLLOWER]: Refusing update from remote peer 132387b04d7e438ab75324b60dbeb8fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:43.844045 28770 raft_consensus.cc:1261] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 1 FOLLOWER]: Refusing update from remote peer 132387b04d7e438ab75324b60dbeb8fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:43.844736 29137 consensus_queue.cc:1035] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2cc4ab5f7dc4f29a1e4a619c153f651" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:43.844923 29150 consensus_queue.cc:1035] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b24d3e0bf4c4fb5ba47d222b9ed61af" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:43.852142 29153 mvcc.cc:205] Tried to move back new op lower bound from 7332799790457798656 to 7332799790350266368. Current Snapshot: MvccSnapshot[applied={T|T < 7332799790457798656}]
I20260924 07:18:43.854967 29154 mvcc.cc:205] Tried to move back new op lower bound from 7332799790457798656 to 7332799790350266368. Current Snapshot: MvccSnapshot[applied={T|T < 7332799790457798656}]
I20260924 07:18:43.863724 28624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:43.863893 28624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:43.865087 29155 mvcc.cc:205] Tried to move back new op lower bound from 7332799790457798656 to 7332799790350266368. Current Snapshot: MvccSnapshot[applied={T|T < 7332799790457798656}]
I20260924 07:18:43.873536 28624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8593d79526d146faa0f95a66b5ebbc05: Sending DeleteTablet for 3 replicas of tablet 3cfde9a7e9a142d887a2d63fa32e0553
I20260924 07:18:43.874289 28758 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cfde9a7e9a142d887a2d63fa32e0553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:43 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:41640
I20260924 07:18:43.874775 28897 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cfde9a7e9a142d887a2d63fa32e0553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:43 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:60516
I20260924 07:18:43.875303 29164 tablet_replica.cc:330] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: stopping tablet replica
I20260924 07:18:43.875355 29028 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cfde9a7e9a142d887a2d63fa32e0553 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:43 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:36248
I20260924 07:18:43.875635 29164 raft_consensus.cc:2229] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:43.876051 29163 tablet_replica.cc:330] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651: stopping tablet replica
I20260924 07:18:43.876194 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28669
I20260924 07:18:43.876395 29163 raft_consensus.cc:2229] T 3cfde9a7e9a142d887a2d63fa32e0553 P d2cc4ab5f7dc4f29a1e4a619c153f651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:43.877535 29164 raft_consensus.cc:2258] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:43.877939 29165 tablet_replica.cc:330] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: stopping tablet replica
I20260924 07:18:43.878224 29165 raft_consensus.cc:2229] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:43.878592 29165 raft_consensus.cc:2258] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:43.879902 29164 ts_tablet_manager.cc:1905] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:43.883466 29164 ts_tablet_manager.cc:1918] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:43.883581 29164 log.cc:1196] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/3cfde9a7e9a142d887a2d63fa32e0553
I20260924 07:18:43.883948 29164 ts_tablet_manager.cc:1939] T 3cfde9a7e9a142d887a2d63fa32e0553 P 7b24d3e0bf4c4fb5ba47d222b9ed61af: Deleting consensus metadata
W20260924 07:18:43.884222 28616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:35541 (error 108)
I20260924 07:18:43.884980 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28833
I20260924 07:18:43.887974 29165 ts_tablet_manager.cc:1905] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:43.890533 29165 ts_tablet_manager.cc:1918] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:43.890637 29165 log.cc:1196] T 3cfde9a7e9a142d887a2d63fa32e0553 P 132387b04d7e438ab75324b60dbeb8fa: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/3cfde9a7e9a142d887a2d63fa32e0553
I20260924 07:18:43.891083 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28966
I20260924 07:18:43.892410 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.892081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:35541 (local address 127.0.0.1:41642)
0924 07:18:43.892255 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.892323 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:35541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 07:18:43.892642 28616 catalog_manager.cc:4584] TS d2cc4ab5f7dc4f29a1e4a619c153f651 (127.15.127.129:35541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3cfde9a7e9a142d887a2d63fa32e0553: Network error: Client connection negotiation failed: client connection to 127.15.127.129:35541: connect: Connection refused (error 111)
I20260924 07:18:43.893592 29125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:43.893341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:42419 (local address 127.0.0.1:60528)
0924 07:18:43.893484 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:43.893531 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:42419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 07:18:43.897035 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 28598
I20260924 07:18:43.906069 15870 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:18:44.241647 15870 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:18:44.241748 15870 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:18:44.586321 15870 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:18:44.586 UTC [27664] LOG: received smart shutdown request
2026-09-24 07:18:44.590 UTC [27664] LOG: background worker "logical replication launcher" (PID 27673) exited with exit code 1
2026-09-24 07:18:44.590 UTC [27668] LOG: shutting down
2026-09-24 07:18:44.751 UTC [27664] LOG: database system is shut down
2026-09-24T07:18:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48462 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:44 dist-test-slave-zx46 krb5kdc[29175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:44 dist-test-slave-zx46 krb5kdc[29175](info): set up 2 sockets
Sep 24 07:18:44 dist-test-slave-zx46 krb5kdc[29175](info): commencing operation
krb5kdc: starting...
W20260924 07:18:45.900907 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:18:45 dist-test-slave-zx46 krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:18:46.024442 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:35817
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:35817,127.15.127.189:42087,127.15.127.188:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:46.142802 29188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.143077 29188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.143147 29188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.147225 29188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:46.147276 29188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.147295 29188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.147311 29188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:46.147325 29188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:46.152479 29188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:35817,127.15.127.189:42087,127.15.127.188:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:46.153623 29188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.154716 29188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:46.162982 29194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.163092 29188 server_base.cc:1034] running on GCE node
W20260924 07:18:46.162999 29193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.162998 29196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.163795 29188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.164786 29188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.165937 29188 hybrid_clock.cc:648] HybridClock initialized: now 1790234326165906 us; error 42 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.168623 29188 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:46.169484 29188 webserver.cc:460] Webserver started at http://127.15.127.190:36719/ using document root <none> and password file <none>
I20260924 07:18:46.169979 29188 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.170035 29188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.170212 29188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:46.172326 29188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "ccfad2eeae2e42f1bd216e429f1facb4"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "b62dfe532abe65ad1aa2fe3105b5239c"
server_key_iv: "d56f7beec850fe0d29be5251a0e29abf"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.172838 29188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "ccfad2eeae2e42f1bd216e429f1facb4"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "b62dfe532abe65ad1aa2fe3105b5239c"
server_key_iv: "d56f7beec850fe0d29be5251a0e29abf"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.176308 29188 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:18:46.178680 29203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.179823 29188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:46.179919 29188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "ccfad2eeae2e42f1bd216e429f1facb4"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "b62dfe532abe65ad1aa2fe3105b5239c"
server_key_iv: "d56f7beec850fe0d29be5251a0e29abf"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.179996 29188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:46.189574 29188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:46.192323 29188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:46.192477 29188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:46.199070 29188 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:35817
I20260924 07:18:46.199337 29255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:35817 every 8 connection(s)
I20260924 07:18:46.200016 29188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:46.202092 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29188
I20260924 07:18:46.202227 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:18:46.202533 15870 external_mini_cluster.cc:1428] Setting key 9c07d47900944f873088d41b2f9f09b6
I20260924 07:18:46.203040 29256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:46.208541 29256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.189@KRBTEST.COM" created.
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:46.224617 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.211342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56360 (local address 127.15.127.190:35817)
0924 07:18:46.211873 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.211892 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.212114 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:18:46.213230 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.213235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.213444 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.213689 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.214472 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.215405 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.216422 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.217136 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.217312 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.219699 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.219707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.219720 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.219745 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.221742 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.222252 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.222256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.222257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.222362 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.222672 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.222675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.222676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.223030 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.223133 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.223572 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.223775 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":146,"threads_started":1}
I20260924 07:18:46.224627 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.211342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35817 (local address 127.0.0.1:56360)
0924 07:18:46.211959 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.211996 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.212846 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.213635 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.213640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.214080 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.214326 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.214339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.215555 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.215558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.216274 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.216283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.217306 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.217309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.217410 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.218070 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.218086 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.219545 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.221877 (+ 2332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.221885 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.221892 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.222140 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.222453 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.222455 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.222457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.222571 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.223141 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.223146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.223328 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.223497 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.223900 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":124,"spinlock_wait_cycles":54016,"thread_start_us":160,"threads_started":1}
I20260924 07:18:46.227234 29256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:18:46.228030 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.227727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58298)
0924 07:18:46.227849 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.227920 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 07:18:46.228235 29204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab.
Entry for principal kudu/127.15.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab.
I20260924 07:18:46.229102 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.228801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58300)
0924 07:18:46.228935 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.229033 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 07:18:46.229375 29256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:42087: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
I20260924 07:18:46.229933 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:42087
--webserver_interface=127.15.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab
--principal=kudu/127.15.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:35817,127.15.127.189:42087,127.15.127.188:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:46.278666 29256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } attempt: 1
I20260924 07:18:46.279654 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.279327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58310)
0924 07:18:46.279499 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.279570 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 07:18:46.280431 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.280224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58314)
0924 07:18:46.280325 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.280378 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:18:46.280586 29256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:42087: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
W20260924 07:18:46.339787 29263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.340080 29263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.340122 29263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.344204 29263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:46.344256 29263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.344274 29263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.344291 29263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:46.344342 29263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:46.348799 29256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } attempt: 2
I20260924 07:18:46.349706 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.349412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58320)
0924 07:18:46.349571 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.349634 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 07:18:46.349713 29263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:35817,127.15.127.189:42087,127.15.127.188:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab
--principal=kudu/127.15.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:42087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:46.350482 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.350253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58330)
0924 07:18:46.350352 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.350424 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:18:46.350626 29256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:42087: Network error: Client connection negotiation failed: client connection to 127.15.127.189:42087: connect: Connection refused (error 111)
I20260924 07:18:46.351069 29263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.352104 29263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:46.360423 29271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.360313 29268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.360469 29263 server_base.cc:1034] running on GCE node
W20260924 07:18:46.360317 29269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.361164 29263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.362301 29263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.363461 29263 hybrid_clock.cc:648] HybridClock initialized: now 1790234326363444 us; error 26 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.366298 29263 init.cc:377] Logged in from keytab as kudu/127.15.127.189@KRBTEST.COM (short username kudu)
I20260924 07:18:46.367164 29263 webserver.cc:460] Webserver started at http://127.15.127.189:36935/ using document root <none> and password file <none>
I20260924 07:18:46.367722 29263 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.367781 29263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.368017 29263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:46.370146 29263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data/instance:
uuid: "93dbd0cce56943e0892a0695dc244c4a"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "4f35136a80e3cb1b649322b37cb6a7db"
server_key_iv: "7bcd1f868d722914cae1e93e1ca54243"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.370710 29263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal/instance:
uuid: "93dbd0cce56943e0892a0695dc244c4a"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "4f35136a80e3cb1b649322b37cb6a7db"
server_key_iv: "7bcd1f868d722914cae1e93e1ca54243"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.374365 29263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:18:46.376703 29278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.377967 29263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:46.378110 29263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal
uuid: "93dbd0cce56943e0892a0695dc244c4a"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "4f35136a80e3cb1b649322b37cb6a7db"
server_key_iv: "7bcd1f868d722914cae1e93e1ca54243"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.378222 29263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:46.391870 29263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:46.394747 29263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:46.394950 29263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:46.402362 29263 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.189:42087
I20260924 07:18:46.402719 29330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.189:42087 every 8 connection(s)
I20260924 07:18:46.403388 29263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
I20260924 07:18:46.405821 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29263
I20260924 07:18:46.405951 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-1/wal/instance
I20260924 07:18:46.406257 15870 external_mini_cluster.cc:1428] Setting key 651f3940aac9e1314eb90899569c8df1
I20260924 07:18:46.406397 29331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:46.410310 29331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.127.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.188@KRBTEST.COM" created.
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:46.425246 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.412295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56368 (local address 127.15.127.190:35817)
0924 07:18:46.412480 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.412487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.413335 (+ 848us) server_negotiation.cc:407] Connection header received
0924 07:18:46.414296 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.414299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.414347 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.414426 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.415778 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.416415 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.418147 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.418580 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.418697 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.421167 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.421173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.421177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.421211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.422943 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.423455 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.423461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.423463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.423567 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.423920 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.423927 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.423931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.424199 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.424319 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.424863 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.425048 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 07:18:46.425981 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.412656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35817 (local address 127.0.0.1:56368)
0924 07:18:46.413152 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.413215 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.414099 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.414650 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.414657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.415088 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.415579 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.415593 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.416567 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.416573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.418025 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.418039 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.418732 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.418735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.418876 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.419493 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.419511 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.420922 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.423064 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.423069 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.423081 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.423333 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.423665 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.423668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.423669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.423806 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.424318 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.424323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.424594 (+ 271us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.424745 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.425046 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":136,"threads_started":1}
I20260924 07:18:46.427661 29331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab.
Entry for principal kudu/127.15.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab.
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:18:46.433743 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:37499
--webserver_interface=127.15.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab
--principal=kudu/127.15.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:35817,127.15.127.189:42087,127.15.127.188:37499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:46.436977 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.428184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58346)
0924 07:18:46.428297 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.428307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.428371 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.428943 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.428946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.429178 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.429281 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.429287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.430350 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.430354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.430950 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.430958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.431604 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.431607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.431745 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.432276 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.432288 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.433626 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.435480 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.435486 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.435488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.435668 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.435940 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.435943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.435945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.436020 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.436478 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.436481 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.436577 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.436686 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.436810 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":23}
I20260924 07:18:46.437084 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.428272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58346 (local address 127.15.127.189:42087)
0924 07:18:46.428568 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.428577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.428619 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:18:46.428662 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.428668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.428827 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.428969 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.429394 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.430233 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.431061 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.431490 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.431607 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.433757 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.433774 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.433786 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.433804 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.435375 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.435770 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.435776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.435778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.435850 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.436110 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.436115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.436117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.436361 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.436505 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.436789 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.436908 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260924 07:18:46.439194 29331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:18:46.439947 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.439711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33348)
0924 07:18:46.439822 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.439887 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 07:18:46.440136 29280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
I20260924 07:18:46.440995 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.440792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33364)
0924 07:18:46.440894 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.440938 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:18:46.441190 29331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:37499: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
I20260924 07:18:46.441784 29256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } attempt: 3
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:18:46.450306 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.442209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58348)
0924 07:18:46.442323 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.442339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.442423 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.442664 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.442666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.442810 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.442911 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.442917 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.444016 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.444018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.444537 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.444546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.445228 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.445230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.445312 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.445802 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.445812 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.447136 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.448945 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.448948 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.448949 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.449192 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.449475 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.449477 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.449478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.449551 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.449941 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.449942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.450017 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.450102 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.450197 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:18:46.450416 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.442254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.15.127.189:42087)
0924 07:18:46.442385 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.442391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.442408 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:18:46.442494 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.442498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.442553 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.442649 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.443031 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.443918 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.444650 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.445121 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.445227 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.447261 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.447265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.447267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.447291 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.448844 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.449298 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.449302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.449304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.449386 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.449632 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.449635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.449637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.449854 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.449947 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.450172 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.450277 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:18:46.451733 29256 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:18:46.452903 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.452689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33378)
0924 07:18:46.452802 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.452845 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 07:18:46.453616 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.453409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33382)
0924 07:18:46.453534 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.453574 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260924 07:18:46.453801 29256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:37499: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
I20260924 07:18:46.484952 29256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } attempt: 1
I20260924 07:18:46.485817 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.485541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33386)
0924 07:18:46.485683 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.485744 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:18:46.486490 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.486314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33392)
0924 07:18:46.486403 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.486443 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:18:46.486630 29256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:37499: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
I20260924 07:18:46.490388 29331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } attempt: 1
I20260924 07:18:46.491063 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.490812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33400)
0924 07:18:46.490947 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.490995 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:18:46.491793 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.491589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33402)
0924 07:18:46.491689 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.491745 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:18:46.491915 29331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:37499: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
W20260924 07:18:46.547768 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:46.548075 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:46.548151 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:46.552147 29337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:46.552246 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:46.552286 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:46.552325 29337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:46.552356 29337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:46.557471 29337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:35817,127.15.127.189:42087,127.15.127.188:37499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab
--principal=kudu/127.15.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:37499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:46.558560 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:46.559599 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:18:46.560073 29331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } attempt: 2
I20260924 07:18:46.560834 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.560570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33408)
0924 07:18:46.560699 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.560765 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 07:18:46.561517 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.561322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33414)
0924 07:18:46.561421 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.561468 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 07:18:46.561641 29331 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:37499: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
I20260924 07:18:46.561790 29256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } attempt: 2
I20260924 07:18:46.562459 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.562228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33426)
0924 07:18:46.562349 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.562397 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:18:46.563076 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.562870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33428)
0924 07:18:46.562970 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.563013 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:18:46.563230 29256 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:37499: Network error: Client connection negotiation failed: client connection to 127.15.127.188:37499: connect: Connection refused (error 111)
W20260924 07:18:46.568357 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.568357 29346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:46.568382 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:46.568676 29337 server_base.cc:1034] running on GCE node
I20260924 07:18:46.569350 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:46.570464 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:46.571666 29337 hybrid_clock.cc:648] HybridClock initialized: now 1790234326571650 us; error 33 us; skew 500 ppm
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:46.574452 29337 init.cc:377] Logged in from keytab as kudu/127.15.127.188@KRBTEST.COM (short username kudu)
I20260924 07:18:46.575357 29337 webserver.cc:460] Webserver started at http://127.15.127.188:38675/ using document root <none> and password file <none>
I20260924 07:18:46.575888 29337 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:46.575945 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:46.576108 29337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:46.578143 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data/instance:
uuid: "4b7a1ec371064a29a45fbbba749d8888"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "e2a3482d73e7e2f06d89703fed597788"
server_key_iv: "4bdc935657f803d3fdc5d41123201c88"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.578646 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal/instance:
uuid: "4b7a1ec371064a29a45fbbba749d8888"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "e2a3482d73e7e2f06d89703fed597788"
server_key_iv: "4bdc935657f803d3fdc5d41123201c88"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.582286 29337 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:18:46.584681 29353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:46.585958 29337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:46.586061 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal
uuid: "4b7a1ec371064a29a45fbbba749d8888"
format_stamp: "Formatted at 2026-09-24 07:18:46 on dist-test-slave-zx46"
server_key: "e2a3482d73e7e2f06d89703fed597788"
server_key_iv: "4bdc935657f803d3fdc5d41123201c88"
server_key_version: "encryptionkey@0"
I20260924 07:18:46.586145 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:46.621181 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:46.624006 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:46.624158 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:46.631011 29337 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.188:37499
I20260924 07:18:46.631294 29405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.188:37499 every 8 connection(s)
I20260924 07:18:46.632042 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
I20260924 07:18:46.634830 29406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:46.638595 29406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:18:46.640841 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29337
I20260924 07:18:46.640990 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790234163642002-15870-0/minicluster-data/master-2/wal/instance
I20260924 07:18:46.641358 15870 external_mini_cluster.cc:1428] Setting key c889620759cdc8da47a35a15c7735da2
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:18:46.652442 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.640650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.15.127.190:35817)
0924 07:18:46.640831 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.640836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.641722 (+ 886us) server_negotiation.cc:407] Connection header received
0924 07:18:46.642864 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.642867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.642911 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.642974 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.643903 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.644396 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.645542 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.645814 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.645889 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.648539 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.648542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.648544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.648566 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.650237 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.650859 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.650861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.650863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.650930 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.651247 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.651250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.651251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.651498 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.651606 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.652154 (+ 548us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.652276 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 07:18:46.652801 29331 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } attempt: 3
I20260924 07:18:46.653478 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.640917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35817 (local address 127.0.0.1:56380)
0924 07:18:46.641542 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.641601 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.642656 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.643025 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.643036 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.643456 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.643750 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.643771 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.644504 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.644507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.645429 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.645438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.645944 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.645948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.646129 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.646816 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.646839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.648380 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.650347 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.650354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.650365 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.650763 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.651007 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.651009 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.651010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.651143 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.651605 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.651610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.651916 (+ 306us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.652097 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.652426 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":159,"threads_started":1}
I20260924 07:18:46.653478 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.642055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.15.127.188:37499)
0924 07:18:46.642447 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.642458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.642495 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:18:46.642655 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.642674 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.642906 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.643090 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.643778 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.644788 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.645669 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.646112 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.646214 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.648937 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.648947 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.648957 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.648976 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.650770 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.651344 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.651349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.651350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.651414 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.651686 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.651689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.651690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.651957 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.652154 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.652385 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.652500 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":128,"threads_started":1}
I20260924 07:18:46.655047 29406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:18:46.662374 29256 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } attempt: 3
I20260924 07:18:46.663342 29334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.653342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33448)
0924 07:18:46.653559 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.653575 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.653655 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.653962 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.653965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.654153 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.654259 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.654266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.655376 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.655381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.656170 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.656180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.656749 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.656751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.656815 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.657271 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.657285 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.658659 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.661368 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.661372 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.661375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.661648 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.662224 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.662227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.662229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.662335 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.662802 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.662806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.662918 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.663008 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.663234 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260924 07:18:46.663923 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.653374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33448 (local address 127.15.127.188:37499)
0924 07:18:46.653729 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.653737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.653751 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:18:46.653783 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.653785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.653830 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.653917 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.654490 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.655240 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.656308 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.656640 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.656737 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.658799 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.658803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.658805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.658827 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.661264 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.662055 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.662058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.662060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.662142 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.662440 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.662446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.662451 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.662677 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.663543 (+ 866us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.663617 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.663733 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:18:46.667984 29331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: Bootstrap starting.
I20260924 07:18:46.668183 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.656128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.15.127.190:35817)
0924 07:18:46.656267 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.656271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.656317 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:18:46.657097 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.657101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.657170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.657249 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.658448 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.659221 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.661737 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.661968 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.662089 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.664838 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.664842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.664844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.664873 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.666600 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.667008 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.667012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.667014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.667097 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.667382 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.667387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.667388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.667579 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.667814 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.667928 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.668030 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:18:46.670938 29331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:46.671620 29331 log.cc:824] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:46.671665 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.662747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33454)
0924 07:18:46.662997 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.663010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.663103 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.663469 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.663472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.663652 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.663757 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.663766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.664631 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.664636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.665410 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.665420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.666098 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.666101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.666172 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.666633 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.666647 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.668003 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.670138 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.670141 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.670143 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.670352 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.670636 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.670641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.670643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.670752 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.671190 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.671193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.671276 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.671366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.671479 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 07:18:46.672158 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.663002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.15.127.188:37499)
0924 07:18:46.663095 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.663099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.663111 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:46.663227 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.663232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.663281 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.663355 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.663883 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.664528 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.665551 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.665910 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.666000 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.668147 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.668150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.668152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.668174 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.670024 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.670455 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.670459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.670461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.670545 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.670865 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.670869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.670871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.671060 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.671147 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.671883 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.672018 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 07:18:46.674132 29331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: No bootstrap required, opened a new log
I20260924 07:18:46.674723 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.655770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.15.127.189:42087)
0924 07:18:46.655873 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.655877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.655889 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:46.655920 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.655923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.655959 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.656035 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.656454 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.657170 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.659426 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.659808 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.659928 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.662068 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.662071 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.662074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.662099 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.672849 (+ 10750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.673360 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.673364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.673365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.673451 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.673778 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.673782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.673787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.674032 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.674108 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.674465 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.674594 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:18:46.675138 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.655539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:42087 (local address 127.0.0.1:58350)
0924 07:18:46.655774 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.655789 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.655854 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.656073 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.656076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.656233 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.656351 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.656358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.658448 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.658451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.659310 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.659321 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.659931 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.659933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.660003 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.660396 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.660406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.661926 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.672954 (+ 11028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.672957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.672959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.673241 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.673580 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.673582 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.673584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.673667 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.674188 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.674193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.674286 (+ 93us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.674850 (+ 564us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.674984 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:18:46.675908 29256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: Bootstrap starting.
I20260924 07:18:46.675895 29406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:18:46.678371 29256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:46.679278 29256 log.cc:824] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:46.679508 29331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.679829 29331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:46.679929 29331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93dbd0cce56943e0892a0695dc244c4a, State: Initialized, Role: FOLLOWER
I20260924 07:18:46.680482 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.669513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.15.127.189:42087)
0924 07:18:46.669869 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.669880 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.669895 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:18:46.669933 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.669936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.669982 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.670091 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.670772 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.671531 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.672871 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.673251 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.673338 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.676444 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.676451 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.676453 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.676476 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.678260 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.678774 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.678777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.678779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.678877 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.679178 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.679182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.679183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.679389 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.679586 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.680199 (+ 613us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.680310 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":100,"threads_started":1}
I20260924 07:18:46.680470 29331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.681934 29331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:46.682808 29256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: No bootstrap required, opened a new log
I20260924 07:18:46.683791 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:46.684326 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.684453 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:46.686940 29256 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.687172 29256 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:46.687217 29256 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfad2eeae2e42f1bd216e429f1facb4, State: Initialized, Role: FOLLOWER
I20260924 07:18:46.687638 29256 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
W20260924 07:18:46.688020 29426 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:46.688102 29426 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:46.688733 29256 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 07:18:46 dist-test-slave-zx46 krb5kdc[29175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234326, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:18:46.687688 29263 master_runner.cc:186] Error getting master registration for 127.15.127.190:35817: OK, instance_id { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" instance_seqno: 1790234326197280 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:18:46.691135 29263 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:18:46.693185 29435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.693406 29435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:18:46.693980 29209 tablet.cc:2369] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:46.694835 29446 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:46.694959 29446 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:46.699009 29410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.676367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.15.127.188:37499)
0924 07:18:46.676458 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.676461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.676866 (+ 405us) server_negotiation.cc:407] Connection header received
0924 07:18:46.677417 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.677420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.677460 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.677504 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.678818 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.679491 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.680487 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.680801 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.680899 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.692126 (+ 11227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.692130 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.692132 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.692155 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.696834 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.697768 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.697773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.697775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.697850 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.698133 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.698136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.698138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.698342 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.698439 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.698742 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.698856 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:46.699409 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.676312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:37499 (local address 127.0.0.1:33458)
0924 07:18:46.676808 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:46.676821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:46.676941 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:46.678424 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:46.678427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:46.678596 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:46.678689 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.678694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.679627 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.679630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.680364 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:46.680373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.680995 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.681020 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:46.681121 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.689434 (+ 8313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:46.689462 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:46.691954 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:46.697387 (+ 5433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.697392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.697397 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.697653 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.697950 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:46.697955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:46.697958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:46.698044 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:46.698489 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:46.698494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:46.698585 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 07:18:46.699147 (+ 562us) client_negotiation.cc:240] Negotiation successful
0924 07:18:46.699274 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428}
I20260924 07:18:46.699412 29259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.686595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56384 (local address 127.15.127.190:35817)
0924 07:18:46.686701 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.686705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.689275 (+ 2570us) server_negotiation.cc:407] Connection header received
0924 07:18:46.689398 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.689401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.689447 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.689515 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.691664 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.692386 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.693892 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.694186 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.694269 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.695913 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.695917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.695920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.695943 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.697647 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.698128 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.698132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.698133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.698207 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.698522 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.698525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.698527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.698749 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.698843 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.699184 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.699283 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:46.704208 29406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888: Bootstrap starting.
I20260924 07:18:46.698611 29188 master_runner.cc:186] Error getting master registration for 127.15.127.188:37499: OK, instance_id { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" instance_seqno: 1790234326629174 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:18:46.705799 29188 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:18:46.707129 29406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:46.708128 29406 log.cc:824] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:46.708384 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.686595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.15.127.189:42087)
0924 07:18:46.686701 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.686705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.695853 (+ 9148us) server_negotiation.cc:407] Connection header received
0924 07:18:46.695961 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.695963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.696004 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.696071 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.700490 (+ 4419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.701279 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.702305 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.702715 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.702821 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.704477 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.704485 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.704488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.704516 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.706306 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.706918 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.706922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.706924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.707028 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.707379 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.707384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.707386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.707612 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.707714 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.708109 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.708230 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:18:46.710645 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:46.686306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.15.127.188:37499)
0924 07:18:46.686403 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:46.686407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:46.692696 (+ 6289us) server_negotiation.cc:407] Connection header received
0924 07:18:46.692864 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:46.692867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:46.692909 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:46.692955 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:46.693908 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.694653 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.697205 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:46.697565 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:46.697646 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:46.706914 (+ 9268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:46.706920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:46.706926 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:46.706973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:46.708626 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.709232 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.709235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.709237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.709338 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:46.709669 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:46.709673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:46.709675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:46.709887 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:46.710007 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:46.710363 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 07:18:46.710491 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:18:46.711880 29406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888: No bootstrap required, opened a new log
I20260924 07:18:46.714860 29406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.715054 29406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:46.715091 29406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b7a1ec371064a29a45fbbba749d8888, State: Initialized, Role: FOLLOWER
I20260924 07:18:46.715515 29406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.715979 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.716096 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:46.716954 29406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 07:18:46.724213 29460 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:18:46.724362 29460 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:18:46.726941 29337 master_runner.cc:263] No leader master found from master 4b7a1ec371064a29a45fbbba749d8888
I20260924 07:18:46.727006 29337 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:18:46.802052 29449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:46.802269 29449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.803704 29449 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccfad2eeae2e42f1bd216e429f1facb4 (127.15.127.190:35817), 93dbd0cce56943e0892a0695dc244c4a (127.15.127.189:42087)
I20260924 07:18:46.804376 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7a1ec371064a29a45fbbba749d8888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dbd0cce56943e0892a0695dc244c4a" is_pre_election: true
I20260924 07:18:46.804694 29305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b7a1ec371064a29a45fbbba749d8888 in term 0.
I20260924 07:18:46.805244 29354 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7a1ec371064a29a45fbbba749d8888, 93dbd0cce56943e0892a0695dc244c4a; no voters:
I20260924 07:18:46.805720 29449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:46.805807 29449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:46.805850 29449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:46.807058 29449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.807390 29449 leader_election.cc:288] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [CANDIDATE]: Term 1 election: Requested vote from peers ccfad2eeae2e42f1bd216e429f1facb4 (127.15.127.190:35817), 93dbd0cce56943e0892a0695dc244c4a (127.15.127.189:42087)
I20260924 07:18:46.807886 29305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7a1ec371064a29a45fbbba749d8888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93dbd0cce56943e0892a0695dc244c4a"
I20260924 07:18:46.808000 29305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:46.810736 29229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7a1ec371064a29a45fbbba749d8888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfad2eeae2e42f1bd216e429f1facb4"
I20260924 07:18:46.810987 29229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:46.812227 29229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7a1ec371064a29a45fbbba749d8888 in term 1.
I20260924 07:18:46.812549 29230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b7a1ec371064a29a45fbbba749d8888" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" is_pre_election: true
I20260924 07:18:46.812661 29230 raft_consensus.cc:2362] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4b7a1ec371064a29a45fbbba749d8888 in term 1. Re-sending same reply.
I20260924 07:18:46.813230 29305 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b7a1ec371064a29a45fbbba749d8888 in term 1.
I20260924 07:18:46.813591 29354 leader_election.cc:302] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b7a1ec371064a29a45fbbba749d8888, 93dbd0cce56943e0892a0695dc244c4a; no voters:
I20260924 07:18:46.815639 29449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:46.816069 29449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [term 1 LEADER]: Becoming Leader. State: Replica: 4b7a1ec371064a29a45fbbba749d8888, State: Running, Role: LEADER
I20260924 07:18:46.816469 29449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } }
I20260924 07:18:46.818295 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7a1ec371064a29a45fbbba749d8888. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.818465 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:46.818878 29464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:46.819924 29464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:46.827538 29305 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [term 1 FOLLOWER]: Refusing update from remote peer 4b7a1ec371064a29a45fbbba749d8888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:46.830127 29449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:46.834496 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7a1ec371064a29a45fbbba749d8888. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.834630 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:46.835721 29230 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [term 1 FOLLOWER]: Refusing update from remote peer 4b7a1ec371064a29a45fbbba749d8888: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:46.836357 29449 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:46.840576 29435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b7a1ec371064a29a45fbbba749d8888. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.840699 29435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:46.850708 29462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.852226 29435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.852351 29435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:46.852806 29462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:46.852996 29465 mvcc.cc:205] Tried to move back new op lower bound from 7332799802674462720 to 7332799802643824640. Current Snapshot: MvccSnapshot[applied={T|T < 7332799802674462720}]
I20260924 07:18:46.854440 29449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.854557 29449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888 [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:46.855908 29415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b7a1ec371064a29a45fbbba749d8888" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccfad2eeae2e42f1bd216e429f1facb4" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35817 } } peers { permanent_uuid: "93dbd0cce56943e0892a0695dc244c4a" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 42087 } } peers { permanent_uuid: "4b7a1ec371064a29a45fbbba749d8888" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 37499 } } }
I20260924 07:18:46.856052 29415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:18:46.856843 29468 mvcc.cc:205] Tried to move back new op lower bound from 7332799802674462720 to 7332799802643824640. Current Snapshot: MvccSnapshot[applied={T|T < 7332799802674462720}]
I20260924 07:18:46.857113 29466 mvcc.cc:205] Tried to move back new op lower bound from 7332799802674462720 to 7332799802643824640. Current Snapshot: MvccSnapshot[applied={T|T < 7332799802674462720}]
I20260924 07:18:46.865159 29464 catalog_manager.cc:1347] Generated new cluster ID: d131ef087b7a481eb710225fa1c23fee
I20260924 07:18:46.865229 29464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:46.881829 29464 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:46.883728 29464 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:46.900707 29464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b7a1ec371064a29a45fbbba749d8888: Generated new TSK 0
I20260924 07:18:46.901520 29464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:47.689613 29426 catalog_manager.cc:1259] Loaded cluster ID: d131ef087b7a481eb710225fa1c23fee
I20260924 07:18:47.689735 29426 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: loading cluster ID for follower catalog manager: success
I20260924 07:18:47.691535 29426 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: acquiring CA information for follower catalog manager: success
I20260924 07:18:47.692616 29426 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 93dbd0cce56943e0892a0695dc244c4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:18:47.696005 29446 catalog_manager.cc:1259] Loaded cluster ID: d131ef087b7a481eb710225fa1c23fee
I20260924 07:18:47.696065 29446 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: loading cluster ID for follower catalog manager: success
I20260924 07:18:47.697655 29446 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: acquiring CA information for follower catalog manager: success
I20260924 07:18:47.698674 29446 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ccfad2eeae2e42f1bd216e429f1facb4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:18:47.965444 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29188
I20260924 07:18:47.971973 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29263
I20260924 07:18:47.977792 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29337
2026-09-24T07:18:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3108 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:48 dist-test-slave-zx46 krb5kdc[29481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:48 dist-test-slave-zx46 krb5kdc[29481](info): set up 2 sockets
Sep 24 07:18:48 dist-test-slave-zx46 krb5kdc[29481](info): commencing operation
krb5kdc: starting...
W20260924 07:18:49.009289 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:18:49 dist-test-slave-zx46 krb5kdc[29481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:18:49.127858 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33775
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:49.235764 29494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:49.236007 29494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:49.236081 29494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:49.239959 29494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:49.240005 29494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:49.240024 29494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:49.240041 29494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:49.240056 29494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:49.245193 29494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:49.246160 29494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:49.247151 29494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:49.255465 29499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.255465 29502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:49.255543 29500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:49.255657 29494 server_base.cc:1034] running on GCE node
I20260924 07:18:49.256394 29494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:49.257507 29494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:49.258697 29494 hybrid_clock.cc:648] HybridClock initialized: now 1790234329258681 us; error 27 us; skew 500 ppm
Sep 24 07:18:49 dist-test-slave-zx46 krb5kdc[29481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:49.261304 29494 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:49.262151 29494 webserver.cc:460] Webserver started at http://127.15.127.190:32925/ using document root <none> and password file <none>
I20260924 07:18:49.262720 29494 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:49.262812 29494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:49.263053 29494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:49.265118 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-zx46"
server_key: "7202fccb6811acd159489d4f3f5a6bb5"
server_key_iv: "21323e37e7371d863989c676ed076e5d"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.265686 29494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-zx46"
server_key: "7202fccb6811acd159489d4f3f5a6bb5"
server_key_iv: "21323e37e7371d863989c676ed076e5d"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.269493 29494 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 07:18:49.271976 29509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:49.273254 29494 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:49.273388 29494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e"
format_stamp: "Formatted at 2026-09-24 07:18:49 on dist-test-slave-zx46"
server_key: "7202fccb6811acd159489d4f3f5a6bb5"
server_key_iv: "21323e37e7371d863989c676ed076e5d"
server_key_version: "encryptionkey@0"
I20260924 07:18:49.273514 29494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:49.286573 29494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:49.289278 29494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:49.289470 29494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:49.296168 29494 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33775
I20260924 07:18:49.296180 29561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33775 every 8 connection(s)
I20260924 07:18:49.297063 29494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:49.299649 29562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:49.303694 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29494
I20260924 07:18:49.303834 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:18:49.304132 15870 external_mini_cluster.cc:1428] Setting key 5828d6e1423b86fb7362b7651570419f
I20260924 07:18:49.305517 29562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e: Bootstrap starting.
I20260924 07:18:49.308142 29562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:49.308889 29562 log.cc:824] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:49.311079 29562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e: No bootstrap required, opened a new log
Sep 24 07:18:49 dist-test-slave-zx46 krb5kdc[29481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:49.315798 29562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33775 } }
I20260924 07:18:49.316104 29562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:49.316174 29562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee5b7b43b3b4295bc9e1fcc18c76a9e, State: Initialized, Role: FOLLOWER
I20260924 07:18:49.316639 29562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33775 } }
I20260924 07:18:49.316812 29562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:49.316882 29562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:49.316985 29562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:49.317837 29562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33775 } }
I20260924 07:18:49.318259 29562 leader_election.cc:302] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ee5b7b43b3b4295bc9e1fcc18c76a9e; no voters:
I20260924 07:18:49.318619 29562 leader_election.cc:288] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:49.318681 29567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:49.318866 29567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [term 1 LEADER]: Becoming Leader. State: Replica: 6ee5b7b43b3b4295bc9e1fcc18c76a9e, State: Running, Role: LEADER
I20260924 07:18:49.319195 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:49.305465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58856 (local address 127.15.127.190:33775)
0924 07:18:49.305903 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:49.305921 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:49.305963 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:18:49.306843 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:49.306865 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:49.307270 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:49.307623 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:49.308370 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.309543 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.310734 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:49.311226 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:49.311357 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:49.314073 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:49.314080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:49.314102 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:49.314140 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:49.316032 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.316550 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.316556 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.316561 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.316697 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:49.317051 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:49.317054 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:49.317056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:49.317478 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:49.317692 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:49.317964 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 07:18:49.318264 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":113,"threads_started":1}
I20260924 07:18:49.319226 29567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33775 } }
I20260924 07:18:49.319854 29562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:49.320916 29568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ee5b7b43b3b4295bc9e1fcc18c76a9e. Latest consensus state: current_term: 1 leader_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33775 } } }
I20260924 07:18:49.321143 29568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:49.321158 29569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ee5b7b43b3b4295bc9e1fcc18c76a9e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33775 } } }
I20260924 07:18:49.321496 29569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:49.321867 29576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:49.322480 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:18:49.324785 29576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:49.329797 29576 catalog_manager.cc:1347] Generated new cluster ID: 12c4d8fbf7c1465f903576b16cb83c3f
I20260924 07:18:49.329865 29576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:49.339324 29576 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:49.340180 29576 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:49.346485 29576 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ee5b7b43b3b4295bc9e1fcc18c76a9e: Generated new TSK 0
I20260924 07:18:49.347074 29576 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:18:49.378732 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29494
2026-09-24T07:18:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:18:49 dist-test-slave-zx46 krb5kdc[29590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:18:49 dist-test-slave-zx46 krb5kdc[29590](info): set up 2 sockets
Sep 24 07:18:49 dist-test-slave-zx46 krb5kdc[29590](info): commencing operation
krb5kdc: starting...
W20260924 07:18:50.409328 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:18:50 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:18:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:18:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:18:50.527361 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33357
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:50.636873 29603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:50.637204 29603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:50.637250 29603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:50.641101 29603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:50.641152 29603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:50.641170 29603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:50.641186 29603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:50.641201 29603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:50.646358 29603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:50.647360 29603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:50.648307 29603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:50.657109 29608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:50.657135 29611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:50.657171 29603 server_base.cc:1034] running on GCE node
W20260924 07:18:50.657111 29609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:50.657955 29603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:50.658990 29603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:50.660161 29603 hybrid_clock.cc:648] HybridClock initialized: now 1790234330660144 us; error 30 us; skew 500 ppm
Sep 24 07:18:50 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234330, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:50.662979 29603 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:50.663938 29603 webserver.cc:460] Webserver started at http://127.15.127.190:37629/ using document root <none> and password file <none>
I20260924 07:18:50.664529 29603 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:50.664597 29603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:50.664809 29603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:50.666975 29603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "2114a796eeb64057be8ded29eca472de"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "50370396b5d7e95f6f12f79130106830"
server_key_iv: "06c5248f3ff9d18a29f66262fc341743"
server_key_version: "encryptionkey@0"
I20260924 07:18:50.667565 29603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "2114a796eeb64057be8ded29eca472de"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "50370396b5d7e95f6f12f79130106830"
server_key_iv: "06c5248f3ff9d18a29f66262fc341743"
server_key_version: "encryptionkey@0"
I20260924 07:18:50.671205 29603 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:18:50.673586 29618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:50.674829 29603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:50.675071 29603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "2114a796eeb64057be8ded29eca472de"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "50370396b5d7e95f6f12f79130106830"
server_key_iv: "06c5248f3ff9d18a29f66262fc341743"
server_key_version: "encryptionkey@0"
I20260924 07:18:50.675196 29603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:50.703816 29603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:50.706956 29603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:50.707165 29603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:50.714172 29603 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33357
I20260924 07:18:50.714475 29670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33357 every 8 connection(s)
I20260924 07:18:50.715176 29603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:50.717883 29671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:50.723238 29671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap starting.
I20260924 07:18:50.723477 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29603
I20260924 07:18:50.723628 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:18:50.723925 15870 external_mini_cluster.cc:1428] Setting key 7a1d29bc9ffdc3754538ddbb1a3a421a
I20260924 07:18:50.725986 29671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:50.726701 29671 log.cc:824] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:50.728488 29671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: No bootstrap required, opened a new log
I20260924 07:18:50.731771 29671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:50.732081 29671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:50.732149 29671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2114a796eeb64057be8ded29eca472de, State: Initialized, Role: FOLLOWER
Sep 24 07:18:50 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:50.732645 29671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:50.732823 29671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:50.732899 29671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:50.733021 29671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:50.733896 29671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:50.734256 29671 leader_election.cc:302] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2114a796eeb64057be8ded29eca472de; no voters:
I20260924 07:18:50.734562 29671 leader_election.cc:288] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:18:50.734669 29676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:50.734884 29676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 LEADER]: Becoming Leader. State: Replica: 2114a796eeb64057be8ded29eca472de, State: Running, Role: LEADER
I20260924 07:18:50.735266 29676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:50.735927 29671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:50.736873 29678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2114a796eeb64057be8ded29eca472de. Latest consensus state: current_term: 1 leader_uuid: "2114a796eeb64057be8ded29eca472de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } } }
I20260924 07:18:50.737028 29678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:50.737349 29677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2114a796eeb64057be8ded29eca472de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } } }
I20260924 07:18:50.737437 29677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:50.738845 29685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:50.739445 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:50.725311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.15.127.190:33357)
0924 07:18:50.725726 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:50.725740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:50.725775 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:18:50.726432 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:50.726451 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:50.726779 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:50.727142 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:50.727930 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:50.728941 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:50.729809 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:50.730270 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:50.730442 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:50.733130 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:50.733135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:50.733148 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:50.733197 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:50.735340 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:50.735837 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:50.735843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:50.735849 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:50.735984 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:50.736325 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:50.736328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:50.736330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:50.736890 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:50.738122 (+ 1232us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:50.738268 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:18:50.738548 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":138,"threads_started":1}
I20260924 07:18:50.741580 29685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:50.746706 29685 catalog_manager.cc:1347] Generated new cluster ID: 5c1fe7223fd84fcfb0dbdc5c27f424e6
I20260924 07:18:50.746766 29685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:50.770392 29685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:18:50.771701 29685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:50.778762 29685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Generated new TSK 0
I20260924 07:18:50.779529 29685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:18:50.815594 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33357
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:50.924695 29697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:50.924932 29697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:50.924973 29697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:50.928895 29697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:50.928949 29697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:50.929072 29697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:18:50.934090 29697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:50.935065 29697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:50.936144 29697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:50.945442 29705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:50.945456 29702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:50.945520 29697 server_base.cc:1034] running on GCE node
W20260924 07:18:50.945462 29703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:50.946064 29697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:50.946708 29697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:50.947901 29697 hybrid_clock.cc:648] HybridClock initialized: now 1790234330947889 us; error 28 us; skew 500 ppm
Sep 24 07:18:50 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234330, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:50.950714 29697 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:18:50.951602 29697 webserver.cc:460] Webserver started at http://127.15.127.129:39999/ using document root <none> and password file <none>
I20260924 07:18:50.952155 29697 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:50.952246 29697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:50.952478 29697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:50.954516 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "08e3b77f9054466493eac57ec9aac1d6"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "c39522d68f8f489e55771a281109523a"
server_key_iv: "37b2e46fe832091d7a773039d3815048"
server_key_version: "encryptionkey@0"
I20260924 07:18:50.955088 29697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "08e3b77f9054466493eac57ec9aac1d6"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "c39522d68f8f489e55771a281109523a"
server_key_iv: "37b2e46fe832091d7a773039d3815048"
server_key_version: "encryptionkey@0"
I20260924 07:18:50.958714 29697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 07:18:50.961110 29712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:50.962332 29697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 07:18:50.962468 29697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "08e3b77f9054466493eac57ec9aac1d6"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "c39522d68f8f489e55771a281109523a"
server_key_iv: "37b2e46fe832091d7a773039d3815048"
server_key_version: "encryptionkey@0"
I20260924 07:18:50.962581 29697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:50.988775 29697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:50.992199 29697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:50.992401 29697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:50.992988 29697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:50.993888 29697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:50.993959 29697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:50.994030 29697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:50.994078 29697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.002975 29697 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:34899
I20260924 07:18:51.002997 29825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:34899 every 8 connection(s)
I20260924 07:18:51.003850 29697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:51.012203 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29697
I20260924 07:18:51.012349 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:18:51.012730 15870 external_mini_cluster.cc:1428] Setting key e9bf08fca5a562b47f5d30023b237810
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234330, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:51.018576 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.005682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:36579 (local address 127.15.127.190:33357)
0924 07:18:51.005820 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.005824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.006677 (+ 853us) server_negotiation.cc:407] Connection header received
0924 07:18:51.008124 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.008127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.008185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.008287 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:51.009588 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.010108 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.011225 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.011454 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.011527 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.014276 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:51.014282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:51.014284 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:51.014313 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:51.016200 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.016718 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.016724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.016727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.016848 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.017319 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.017325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.017330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.017583 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:51.017687 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.018208 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.018357 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:18:51.019294 29828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.006079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:36579)
0924 07:18:51.006532 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.006575 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.007897 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.008458 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.008465 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.008817 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:51.009370 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.009410 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.010240 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.010243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.011122 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.011131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.011575 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.011577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.011710 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.012244 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:51.012269 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:51.014128 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:51.016331 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:51.016335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:51.016347 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:51.016605 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:51.016984 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:51.016987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:51.016988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:51.017207 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:51.017690 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:51.017695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:51.017926 (+ 231us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.018076 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.018316 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260924 07:18:51.020541 29826 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33357
I20260924 07:18:51.020804 29826 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:51.021448 29826 heartbeater.cc:511] Master 127.15.127.190:33357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:18:51.023084 29635 ts_manager.cc:194] Registered new tserver with Master: 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899)
I20260924 07:18:51.024307 29635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:36579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:18:51.039907 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33357
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:51.146821 29831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:51.147075 29831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:51.147114 29831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:51.151000 29831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:51.151053 29831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:51.151152 29831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:18:51.156419 29831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:51.157529 29831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:51.158589 29831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:51.168023 29836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.168028 29839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.168183 29837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:51.168278 29831 server_base.cc:1034] running on GCE node
I20260924 07:18:51.168788 29831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:51.169538 29831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:51.170740 29831 hybrid_clock.cc:648] HybridClock initialized: now 1790234331170725 us; error 27 us; skew 500 ppm
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:51.173594 29831 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:18:51.174463 29831 webserver.cc:460] Webserver started at http://127.15.127.130:46817/ using document root <none> and password file <none>
I20260924 07:18:51.175043 29831 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:51.175143 29831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:51.175390 29831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:51.177518 29831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "d97fad6898c943dda8be46b5f5b6e5ae"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "375ba514145a57dcd9298a8ae5205342"
server_key_iv: "ddb089eb6cd106a87669a8580e3a841f"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.178114 29831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "d97fad6898c943dda8be46b5f5b6e5ae"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "375ba514145a57dcd9298a8ae5205342"
server_key_iv: "ddb089eb6cd106a87669a8580e3a841f"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.182111 29831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260924 07:18:51.184551 29846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.186026 29831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:18:51.186165 29831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "d97fad6898c943dda8be46b5f5b6e5ae"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "375ba514145a57dcd9298a8ae5205342"
server_key_iv: "ddb089eb6cd106a87669a8580e3a841f"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.186276 29831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:51.196980 29831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:51.199688 29831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:51.199880 29831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:51.200464 29831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:51.201308 29831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:51.201387 29831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.201457 29831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:51.201501 29831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.209829 29831 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:32825
I20260924 07:18:51.209920 29959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:32825 every 8 connection(s)
I20260924 07:18:51.210750 29831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:51.215574 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29831
I20260924 07:18:51.215714 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:18:51.215983 15870 external_mini_cluster.cc:1428] Setting key 1d718f3e3e707df6f303a0a0cf0a7968
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:51.225235 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.212598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:45549 (local address 127.15.127.190:33357)
0924 07:18:51.212743 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.212746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.213515 (+ 769us) server_negotiation.cc:407] Connection header received
0924 07:18:51.214897 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.214900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.214962 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.215073 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:51.216313 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.216958 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.218202 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.218494 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.218635 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.221146 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:51.221150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:51.221153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:51.221184 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:51.222964 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.223541 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.223546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.223548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.223642 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.223980 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.223983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.223987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.224243 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:51.224351 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.224837 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.224986 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:18:51.226003 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.212893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:45549)
0924 07:18:51.213364 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.213404 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.214682 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.215237 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.215244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.215591 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:51.216173 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.216187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.217135 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.217138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.218070 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.218080 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.218676 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.218679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.218861 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.219473 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:51.219488 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:51.220918 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:51.223098 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:51.223102 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:51.223115 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:51.223436 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:51.223736 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:51.223738 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:51.223739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:51.223864 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:51.224353 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:51.224358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:51.224591 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.224748 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.224995 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":150,"threads_started":1}
I20260924 07:18:51.227234 29960 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33357
I20260924 07:18:51.227505 29960 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:51.227973 29960 heartbeater.cc:511] Master 127.15.127.190:33357 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:18:51.229089 29635 ts_manager.cc:194] Registered new tserver with Master: d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
I20260924 07:18:51.229915 29635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:45549
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:18:51.243752 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33357
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:51.352818 29965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:51.353078 29965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:51.353137 29965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:51.356923 29965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:51.356969 29965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:51.357039 29965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:18:51.361987 29965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:51.362905 29965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:51.364060 29965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:51.373322 29970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.373342 29971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.373524 29973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:51.373589 29965 server_base.cc:1034] running on GCE node
I20260924 07:18:51.374054 29965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:51.374683 29965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:51.375873 29965 hybrid_clock.cc:648] HybridClock initialized: now 1790234331375860 us; error 32 us; skew 500 ppm
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:51.378536 29965 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:18:51.379469 29965 webserver.cc:460] Webserver started at http://127.15.127.131:39519/ using document root <none> and password file <none>
I20260924 07:18:51.380107 29965 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:51.380191 29965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:51.380411 29965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:18:51.382458 29965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "efee6e8cbb844685bba0020a16e76d1c"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "a26e99c36b21686299a7128286af27ff"
server_key_iv: "da34e43338f3f36841129c2b22c6894c"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.383023 29965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "efee6e8cbb844685bba0020a16e76d1c"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "a26e99c36b21686299a7128286af27ff"
server_key_iv: "da34e43338f3f36841129c2b22c6894c"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.386628 29965 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:18:51.388921 29980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.390224 29965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:51.390355 29965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "efee6e8cbb844685bba0020a16e76d1c"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "a26e99c36b21686299a7128286af27ff"
server_key_iv: "da34e43338f3f36841129c2b22c6894c"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.390439 29965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:51.415985 29965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:51.419301 29965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:51.419452 29965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:51.420055 29965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:51.420866 29965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:51.420912 29965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.420951 29965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:51.420990 29965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.429751 29965 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:39229
I20260924 07:18:51.429780 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:39229 every 8 connection(s)
I20260924 07:18:51.430636 29965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:51.440012 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 29965
I20260924 07:18:51.440126 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:51.440474 15870 external_mini_cluster.cc:1428] Setting key 8844b3e9410b4248b38d38a8ac850dd5
I20260924 07:18:51.444587 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.432394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:50593 (local address 127.15.127.190:33357)
0924 07:18:51.432569 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.432574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.433409 (+ 835us) server_negotiation.cc:407] Connection header received
0924 07:18:51.434705 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.434708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.434754 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.434827 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:51.436026 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.436689 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.438011 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.438239 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.438353 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.440958 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:51.440961 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:51.440963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:51.440988 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:51.442755 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.443267 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.443270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.443271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.443351 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.443640 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.443642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.443643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.443810 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:51.443884 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.444354 (+ 470us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.444474 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:18:51.445442 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.432790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:50593)
0924 07:18:51.433266 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.433307 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.434503 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.434985 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.434991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.435333 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:51.435827 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.435900 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.436851 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.436855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.437888 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.437899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.438392 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.438396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.438579 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.439108 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:51.439123 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:51.440740 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:51.442881 (+ 2141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:51.442887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:51.442903 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:51.443179 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:51.443427 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:51.443429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:51.443431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:51.443561 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:51.443906 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:51.443910 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:51.444138 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.444310 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.444560 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":159,"threads_started":1}
I20260924 07:18:51.446554 30094 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33357
I20260924 07:18:51.446830 30094 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:51.447348 30094 heartbeater.cc:511] Master 127.15.127.190:33357 requested a full tablet report, sending...
I20260924 07:18:51.448369 29635 ts_manager.cc:194] Registered new tserver with Master: efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229)
I20260924 07:18:51.448966 29635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:50593
I20260924 07:18:51.454599 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:51.463796 29674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.456487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.15.127.190:33357)
0924 07:18:51.456630 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.456634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.456824 (+ 190us) server_negotiation.cc:407] Connection header received
0924 07:18:51.456914 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.456916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.456957 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.457134 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:51.457865 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.458361 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.459153 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.459353 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.459425 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.460792 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:51.460795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:51.460797 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:51.460821 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:51.462269 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.462657 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.462660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.462661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.462739 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:51.462961 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:51.462963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:51.462964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:51.463174 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:51.463283 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.463572 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.463672 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 07:18:51.466678 29635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:51.469826 29635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:51.484588 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.478725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:39229 (local address 127.0.0.1:44352)
0924 07:18:51.479485 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.479500 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.479597 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.480300 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.480305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.480325 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.480597 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.480603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.481685 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.481691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.482912 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.482922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.484027 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.484031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.484138 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.484179 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.484297 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.484423 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":152,"thread_start_us":110,"threads_started":1}
I20260924 07:18:51.484647 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.479381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.15.127.131:39229)
0924 07:18:51.479824 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.479837 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.479862 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:18:51.479943 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.479948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.480149 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.480452 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.480709 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.481557 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.483061 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.483897 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.484018 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.484154 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.484356 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.484457 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":97,"threads_started":1}
I20260924 07:18:51.485227 30108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.478951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34899 (local address 127.0.0.1:34744)
0924 07:18:51.479574 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.479589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.479665 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.480586 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.480588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.480606 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.480702 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.480707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.482254 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.482257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.483542 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.483553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.484675 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.484679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.484793 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.484823 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.484877 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.484969 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":77,"thread_start_us":92,"threads_started":1}
I20260924 07:18:51.485460 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.478732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.15.127.130:32825)
0924 07:18:51.479736 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.479748 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.479777 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:18:51.479840 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.479844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.480083 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.480759 (+ 676us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.480925 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.482108 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.483610 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.484311 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.484428 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.484544 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.485173 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.485293 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":94,"threads_started":1}
I20260924 07:18:51.485589 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.479748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.15.127.129:34899)
0924 07:18:51.480058 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.480074 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.480097 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:18:51.480170 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.480229 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.480402 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.480826 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.480977 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.482108 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.483681 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.484508 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.485165 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.485300 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.485357 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.485440 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
I20260924 07:18:51.485064 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.478566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:32825 (local address 127.0.0.1:60344)
0924 07:18:51.479235 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.479290 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.479421 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.480256 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.480262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.480286 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.480454 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.480462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.482293 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.482296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.483492 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.483501 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.484608 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.484613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.484710 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.484748 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.484834 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.484907 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":90,"thread_start_us":86,"threads_started":1}
I20260924 07:18:51.488212 30028 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4fdae23444c59969140d262860700 (DEFAULT_TABLE table=test-table [id=17f21fd644a946568a0516576f21655a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:51.489269 30028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4fdae23444c59969140d262860700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:51.489050 29894 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4fdae23444c59969140d262860700 (DEFAULT_TABLE table=test-table [id=17f21fd644a946568a0516576f21655a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:51.489063 29760 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4fdae23444c59969140d262860700 (DEFAULT_TABLE table=test-table [id=17f21fd644a946568a0516576f21655a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:51.490099 29894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4fdae23444c59969140d262860700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:51.490099 29760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4fdae23444c59969140d262860700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:51.495605 30111 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Bootstrap starting.
I20260924 07:18:51.497087 30112 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Bootstrap starting.
I20260924 07:18:51.497504 30113 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap starting.
I20260924 07:18:51.497958 30111 tablet_bootstrap.cc:654] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:51.499259 30111 log.cc:824] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:51.499707 30113 tablet_bootstrap.cc:654] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:51.500974 30111 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: No bootstrap required, opened a new log
I20260924 07:18:51.500959 30113 log.cc:824] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:51.501216 30111 ts_tablet_manager.cc:1397] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260924 07:18:51.501984 30112 tablet_bootstrap.cc:654] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:51.502645 30112 log.cc:824] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:51.504540 30113 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: No bootstrap required, opened a new log
I20260924 07:18:51.504565 30112 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: No bootstrap required, opened a new log
I20260924 07:18:51.504402 30111 raft_consensus.cc:348] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.504642 30111 raft_consensus.cc:374] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:51.504721 30111 raft_consensus.cc:729] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efee6e8cbb844685bba0020a16e76d1c, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.504763 30112 ts_tablet_manager.cc:1397] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260924 07:18:51.504745 30113 ts_tablet_manager.cc:1397] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:18:51.505306 30111 consensus_queue.cc:260] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.505995 30094 heartbeater.cc:503] Master 127.15.127.190:33357 was elected leader, sending a full tablet report...
I20260924 07:18:51.506280 30111 ts_tablet_manager.cc:1428] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260924 07:18:51.508175 30113 raft_consensus.cc:348] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.508378 30113 raft_consensus.cc:374] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:51.508407 30113 raft_consensus.cc:729] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e3b77f9054466493eac57ec9aac1d6, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.508440 30112 raft_consensus.cc:348] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.508709 30112 raft_consensus.cc:374] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:51.508769 30112 raft_consensus.cc:729] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97fad6898c943dda8be46b5f5b6e5ae, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.508802 30113 consensus_queue.cc:260] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.509197 30112 consensus_queue.cc:260] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.509805 30113 ts_tablet_manager.cc:1428] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 07:18:51.509876 29960 heartbeater.cc:503] Master 127.15.127.190:33357 was elected leader, sending a full tablet report...
I20260924 07:18:51.510084 29826 heartbeater.cc:503] Master 127.15.127.190:33357 was elected leader, sending a full tablet report...
I20260924 07:18:51.510090 30112 ts_tablet_manager.cc:1428] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:18:51.546885 30119 raft_consensus.cc:482] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:51.547138 30119 raft_consensus.cc:504] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.548307 30119 leader_election.cc:288] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899)
I20260924 07:18:51.553339 30109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.548788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:36241 (local address 127.15.127.131:39229)
0924 07:18:51.548931 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.548938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.549330 (+ 392us) server_negotiation.cc:407] Connection header received
0924 07:18:51.549400 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.549402 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.549469 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.549556 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.549797 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.550502 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.551948 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.552590 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.552723 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.552781 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.553132 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.553209 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:18:51.553339 30110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.548715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:38955 (local address 127.15.127.129:34899)
0924 07:18:51.548850 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.548855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.548893 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:18:51.549032 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.549036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.549095 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.549193 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:51.549491 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.550342 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.551948 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.552599 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.552723 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.552792 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.553130 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.553209 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:18:51.553361 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.548650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34899 (local address 127.15.127.130:38955)
0924 07:18:51.548766 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.548783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.548898 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.549222 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.549228 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.549250 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.549380 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.549386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.550447 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.550449 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.551793 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.551804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.552724 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.552728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.552943 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.552985 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.553079 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.553202 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":142}
I20260924 07:18:51.553370 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.548652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:39229 (local address 127.15.127.130:36241)
0924 07:18:51.549206 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.549217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:51.549300 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:51.549570 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:51.549572 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:51.549586 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:51.549663 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.549667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.550595 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.550598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.551793 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:51.551804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.552724 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.552728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:51.552938 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.552980 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:18:51.553079 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:18:51.553202 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":333,"thread_start_us":135,"threads_started":1}
I20260924 07:18:51.554152 29780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4fdae23444c59969140d262860700" candidate_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e3b77f9054466493eac57ec9aac1d6" is_pre_election: true
I20260924 07:18:51.554154 30048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4fdae23444c59969140d262860700" candidate_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efee6e8cbb844685bba0020a16e76d1c" is_pre_election: true
I20260924 07:18:51.554466 29780 raft_consensus.cc:2454] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97fad6898c943dda8be46b5f5b6e5ae in term 0.
I20260924 07:18:51.554492 30048 raft_consensus.cc:2454] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d97fad6898c943dda8be46b5f5b6e5ae in term 0.
I20260924 07:18:51.554988 29850 leader_election.cc:302] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6, d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c; no voters:
I20260924 07:18:51.555277 30119 raft_consensus.cc:2781] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:51.555372 30119 raft_consensus.cc:482] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:51.555433 30119 raft_consensus.cc:3037] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:51.556576 30119 raft_consensus.cc:504] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.556923 30119 leader_election.cc:288] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [CANDIDATE]: Term 1 election: Requested vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899)
I20260924 07:18:51.557399 30048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4fdae23444c59969140d262860700" candidate_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efee6e8cbb844685bba0020a16e76d1c"
I20260924 07:18:51.557402 29780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4fdae23444c59969140d262860700" candidate_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e3b77f9054466493eac57ec9aac1d6"
I20260924 07:18:51.557540 29780 raft_consensus.cc:3037] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:51.557571 30048 raft_consensus.cc:3037] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:51.558931 29780 raft_consensus.cc:2454] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97fad6898c943dda8be46b5f5b6e5ae in term 1.
I20260924 07:18:51.558931 30048 raft_consensus.cc:2454] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d97fad6898c943dda8be46b5f5b6e5ae in term 1.
I20260924 07:18:51.559266 29849 leader_election.cc:302] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6, d97fad6898c943dda8be46b5f5b6e5ae; no voters:
I20260924 07:18:51.559441 30119 raft_consensus.cc:2781] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:51.559691 30119 raft_consensus.cc:686] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 LEADER]: Becoming Leader. State: Replica: d97fad6898c943dda8be46b5f5b6e5ae, State: Running, Role: LEADER
I20260924 07:18:51.560029 30119 consensus_queue.cc:237] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:51.563199 29634 catalog_manager.cc:5495] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae reported cstate change: term changed from 0 to 1, leader changed from <none> to d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130). New cstate: current_term: 1 leader_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:51.585474 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.582340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.15.127.130:32825)
0924 07:18:51.582462 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:51.582466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:51.582479 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:18:51.582554 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:51.582556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:51.582610 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:51.582709 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:51.582903 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.583391 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.584165 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:51.584365 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:51.584437 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:51.584674 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:51.585175 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:51.585264 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:51.585322 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 07:18:51.585368 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:18:51.591261 29780 raft_consensus.cc:1261] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Refusing update from remote peer d97fad6898c943dda8be46b5f5b6e5ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:51.591260 30048 raft_consensus.cc:1261] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Refusing update from remote peer d97fad6898c943dda8be46b5f5b6e5ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:51.591945 30119 consensus_queue.cc:1035] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:51.592151 30121 consensus_queue.cc:1035] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:51.602522 30125 mvcc.cc:205] Tried to move back new op lower bound from 7332799822190477312 to 7332799822072389632. Current Snapshot: MvccSnapshot[applied={T|T < 7332799822190477312}]
I20260924 07:18:51.602723 30124 mvcc.cc:205] Tried to move back new op lower bound from 7332799822190477312 to 7332799822072389632. Current Snapshot: MvccSnapshot[applied={T|T < 7332799822190477312 or (T in {7332799822190477312})}]
I20260924 07:18:51.608843 29634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:51.608992 29634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:51.611407 29634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Sending DeleteTablet for 3 replicas of tablet d1f4fdae23444c59969140d262860700
I20260924 07:18:51.612144 29894 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4fdae23444c59969140d262860700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:60344
I20260924 07:18:51.612319 30028 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4fdae23444c59969140d262860700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:44352
I20260924 07:18:51.612435 29760 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4fdae23444c59969140d262860700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:34744
I20260924 07:18:51.612736 30134 tablet_replica.cc:330] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: stopping tablet replica
I20260924 07:18:51.612764 30133 tablet_replica.cc:330] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: stopping tablet replica
I20260924 07:18:51.613058 30133 raft_consensus.cc:2229] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:51.613059 30134 raft_consensus.cc:2229] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:51.613086 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29697
I20260924 07:18:51.613837 30134 raft_consensus.cc:2258] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:51.614277 30133 raft_consensus.cc:2258] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:51.615496 30134 ts_tablet_manager.cc:1905] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:51.615688 30133 ts_tablet_manager.cc:1905] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:51.618244 30133 ts_tablet_manager.cc:1918] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:51.618244 30134 ts_tablet_manager.cc:1918] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:51.618341 30134 log.cc:1196] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/d1f4fdae23444c59969140d262860700
I20260924 07:18:51.618366 30133 log.cc:1196] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/d1f4fdae23444c59969140d262860700
I20260924 07:18:51.618718 30133 ts_tablet_manager.cc:1939] T d1f4fdae23444c59969140d262860700 P d97fad6898c943dda8be46b5f5b6e5ae: Deleting consensus metadata
I20260924 07:18:51.618718 30134 ts_tablet_manager.cc:1939] T d1f4fdae23444c59969140d262860700 P efee6e8cbb844685bba0020a16e76d1c: Deleting consensus metadata
I20260924 07:18:51.619833 29619 catalog_manager.cc:4841] TS d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): tablet d1f4fdae23444c59969140d262860700 (table test-table [id=17f21fd644a946568a0516576f21655a]) successfully deleted
I20260924 07:18:51.619833 29622 catalog_manager.cc:4841] TS efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): tablet d1f4fdae23444c59969140d262860700 (table test-table [id=17f21fd644a946568a0516576f21655a]) successfully deleted
W20260924 07:18:51.620678 29621 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:34899 (error 108)
I20260924 07:18:51.621555 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29831
I20260924 07:18:51.621730 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:51.621448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34899 (local address 127.0.0.1:34754)
0924 07:18:51.621590 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:51.621657 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 07:18:51.622278 29621 catalog_manager.cc:4584] TS 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1f4fdae23444c59969140d262860700: Network error: Client connection negotiation failed: client connection to 127.15.127.129:34899: connect: Connection refused (error 111)
I20260924 07:18:51.627283 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29965
I20260924 07:18:51.632740 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 29603
I20260924 07:18:51.638883 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33357
--webserver_interface=127.15.127.190
--webserver_port=37629
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:51.750865 30137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:51.751125 30137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:51.751192 30137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:51.755321 30137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:51.755431 30137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:51.755489 30137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:51.755548 30137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:51.755577 30137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:51.760823 30137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=37629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:51.761859 30137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:51.762861 30137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:51.775985 30143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.775991 30146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:51.776038 30137 server_base.cc:1034] running on GCE node
W20260924 07:18:51.775982 30144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:51.776891 30137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:51.778105 30137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:51.779263 30137 hybrid_clock.cc:648] HybridClock initialized: now 1790234331779225 us; error 50 us; skew 500 ppm
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:51.782151 30137 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:51.783109 30137 webserver.cc:460] Webserver started at http://127.15.127.190:37629/ using document root <none> and password file <none>
I20260924 07:18:51.783687 30137 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:51.783797 30137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:51.787618 30137 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:51.789738 30153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.790963 30137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:51.791115 30137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "2114a796eeb64057be8ded29eca472de"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "50370396b5d7e95f6f12f79130106830"
server_key_iv: "06c5248f3ff9d18a29f66262fc341743"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.791708 30137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:51.805881 30137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:51.808706 30137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:51.808916 30137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:51.816190 30137 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33357
I20260924 07:18:51.816496 30205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33357 every 8 connection(s)
I20260924 07:18:51.817396 30137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:51.820861 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30137
I20260924 07:18:51.821355 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:34899
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:33357
--webserver_port=39999
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:51.821990 30206 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:18:51.823669 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap starting.
I20260924 07:18:51.840276 30206 log.cc:824] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:51.848668 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:51.849113 30206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap complete.
I20260924 07:18:51.853189 30206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:51.853624 30206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2114a796eeb64057be8ded29eca472de, State: Initialized, Role: FOLLOWER
I20260924 07:18:51.854143 30206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:51.854293 30206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:51.854387 30206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:51.854507 30206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:18:51.856600 30206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:51.856925 30206 leader_election.cc:302] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2114a796eeb64057be8ded29eca472de; no voters:
I20260924 07:18:51.857249 30206 leader_election.cc:288] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:18:51.857395 30210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:18:51.857641 30210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 LEADER]: Becoming Leader. State: Replica: 2114a796eeb64057be8ded29eca472de, State: Running, Role: LEADER
I20260924 07:18:51.857995 30210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:51.858470 30206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:51.858999 30210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2114a796eeb64057be8ded29eca472de. Latest consensus state: current_term: 2 leader_uuid: "2114a796eeb64057be8ded29eca472de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } } }
I20260924 07:18:51.859144 30210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:51.859498 30215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:51.859525 30211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2114a796eeb64057be8ded29eca472de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } } }
I20260924 07:18:51.859717 30211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:51.864192 30215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:51.865074 30215 catalog_manager.cc:1259] Loaded cluster ID: 5c1fe7223fd84fcfb0dbdc5c27f424e6
I20260924 07:18:51.865157 30215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:18:51.866781 30215 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 07:18:51.871138 30215 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 07:18:51.876362 30215 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:18:51.877156 30215 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Loaded TSK: 0
I20260924 07:18:51.878371 30215 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:18:51.936695 30208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:51.936945 30208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:51.936988 30208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:51.941009 30208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:51.941083 30208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:51.941191 30208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:18:51.946290 30208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:34899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=39999
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:51.947285 30208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:51.948256 30208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:51.958739 30233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.958843 30234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:51.958948 30236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:51.959606 30208 server_base.cc:1034] running on GCE node
I20260924 07:18:51.960000 30208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:51.960706 30208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:51.961884 30208 hybrid_clock.cc:648] HybridClock initialized: now 1790234331961850 us; error 56 us; skew 500 ppm
Sep 24 07:18:51 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:51.964741 30208 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:18:51.965679 30208 webserver.cc:460] Webserver started at http://127.15.127.129:39999/ using document root <none> and password file <none>
I20260924 07:18:51.966265 30208 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:51.966341 30208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:51.970203 30208 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260924 07:18:51.972352 30243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:51.973729 30208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:18:51.973821 30208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "08e3b77f9054466493eac57ec9aac1d6"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "c39522d68f8f489e55771a281109523a"
server_key_iv: "37b2e46fe832091d7a773039d3815048"
server_key_version: "encryptionkey@0"
I20260924 07:18:51.974371 30208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:51.994743 30208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:51.997992 30208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:51.998200 30208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:51.998848 30208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:51.999966 30250 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:18:52.003950 30208 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:18:52.004007 30208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 07:18:52.004053 30208 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:18:52.005648 30208 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:18:52.005703 30208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:52.005939 30250 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap starting.
I20260924 07:18:52.016494 30208 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:34899
I20260924 07:18:52.016593 30357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:34899 every 8 connection(s)
I20260924 07:18:52.017719 30208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:52.027814 30250 log.cc:824] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:52.027940 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30208
I20260924 07:18:52.028434 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:32825
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:33357
--webserver_port=46817
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:52.037117 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.021101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:36045 (local address 127.15.127.190:33357)
0924 07:18:52.023105 (+ 2004us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.023119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.023178 (+ 59us) server_negotiation.cc:407] Connection header received
0924 07:18:52.023479 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.023494 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.023905 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.024231 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:52.025804 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.026963 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.028507 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.029211 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.029385 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.032009 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:52.032017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:52.032043 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:52.032069 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:52.033948 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.034500 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.034506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.034511 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.034638 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.034975 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.034980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.034982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.035321 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:52.035460 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.035976 (+ 516us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.036237 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1875,"thread_start_us":126,"threads_started":1}
I20260924 07:18:52.037125 30361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.021113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:36045)
0924 07:18:52.021699 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.021752 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.022568 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.024309 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.024317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.024808 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:52.025583 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.025602 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.027129 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.027132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.028363 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.028372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.029375 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.029381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.029618 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.030346 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:52.030362 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:52.031819 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:52.034096 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.034102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.034115 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.034381 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.034739 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.034744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.034746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.034879 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.035446 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:52.035452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:52.035688 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.035869 (+ 181us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.036198 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":174,"threads_started":1}
I20260924 07:18:52.037492 30250 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:52.038082 30250 tablet_bootstrap.cc:492] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap complete.
I20260924 07:18:52.039628 30250 ts_tablet_manager.cc:1397] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.004s
I20260924 07:18:52.039783 30358 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33357
I20260924 07:18:52.040016 30358 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:52.040467 30358 heartbeater.cc:511] Master 127.15.127.190:33357 requested a full tablet report, sending...
I20260924 07:18:52.042758 30170 ts_manager.cc:194] Registered new tserver with Master: 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899)
I20260924 07:18:52.042673 30250 raft_consensus.cc:348] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.043181 30250 raft_consensus.cc:729] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e3b77f9054466493eac57ec9aac1d6, State: Initialized, Role: FOLLOWER
I20260924 07:18:52.043653 30250 consensus_queue.cc:260] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.044636 30250 ts_tablet_manager.cc:1428] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:18:52.046293 30170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:36045
I20260924 07:18:52.048715 30358 heartbeater.cc:503] Master 127.15.127.190:33357 was elected leader, sending a full tablet report...
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234331, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:18:52.056739 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.045829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34899 (local address 127.0.0.1:34768)
0924 07:18:52.046218 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.046259 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.046425 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.047569 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.047575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.048016 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:52.048151 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.048159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.049458 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.049462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.050067 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.050076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.050867 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.050870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.050947 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.051506 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:52.051518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:52.052811 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:52.054867 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.054874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.054878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.055158 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.055482 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.055487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.055489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.055575 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.056075 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:52.056081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:52.056225 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.056379 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.056566 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
I20260924 07:18:52.056774 30366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.045916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34768 (local address 127.15.127.129:34899)
0924 07:18:52.047112 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.047121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.047139 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:18:52.047189 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.047192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.047386 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.047519 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:52.048286 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.049342 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.050211 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.050741 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.050843 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.052945 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:52.052955 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:52.052967 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:52.052992 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:52.054767 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.055291 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.055297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.055299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.055389 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.055667 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.055672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.055674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.055956 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:52.056094 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.056461 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.056627 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1101,"thread_start_us":81,"threads_started":1}
I20260924 07:18:52.057668 30292 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4fdae23444c59969140d262860700 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:51 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:34768
I20260924 07:18:52.058478 30367 tablet_replica.cc:330] stopping tablet replica
I20260924 07:18:52.058799 30367 raft_consensus.cc:2229] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:52.059063 30367 raft_consensus.cc:2258] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:52.060088 30367 ts_tablet_manager.cc:1905] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:52.063620 30367 ts_tablet_manager.cc:1918] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:52.063701 30367 log.cc:1196] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/d1f4fdae23444c59969140d262860700
I20260924 07:18:52.063987 30367 ts_tablet_manager.cc:1939] T d1f4fdae23444c59969140d262860700 P 08e3b77f9054466493eac57ec9aac1d6: Deleting consensus metadata
I20260924 07:18:52.065616 30156 catalog_manager.cc:4841] TS 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899): tablet d1f4fdae23444c59969140d262860700 (table test-table [id=17f21fd644a946568a0516576f21655a]) successfully deleted
W20260924 07:18:52.145954 30363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:52.146243 30363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:52.146315 30363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:52.150362 30363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:52.150466 30363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:52.150563 30363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:18:52.156040 30363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:32825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=46817
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:52.157219 30363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:52.158272 30363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:52.167945 30375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:52.167964 30372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.168131 30363 server_base.cc:1034] running on GCE node
W20260924 07:18:52.167953 30373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.168624 30363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:52.169277 30363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:52.170457 30363 hybrid_clock.cc:648] HybridClock initialized: now 1790234332170442 us; error 26 us; skew 500 ppm
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:52.173177 30363 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:18:52.173939 30363 webserver.cc:460] Webserver started at http://127.15.127.130:46817/ using document root <none> and password file <none>
I20260924 07:18:52.174427 30363 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:52.174494 30363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:52.178120 30363 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 07:18:52.180117 30382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.181298 30363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:52.181408 30363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "d97fad6898c943dda8be46b5f5b6e5ae"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "375ba514145a57dcd9298a8ae5205342"
server_key_iv: "ddb089eb6cd106a87669a8580e3a841f"
server_key_version: "encryptionkey@0"
I20260924 07:18:52.181945 30363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:52.199618 30363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:52.202562 30363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:52.202713 30363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:52.203217 30363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:52.203969 30363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:52.204013 30363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.204053 30363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:52.204069 30363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.212671 30363 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:32825
I20260924 07:18:52.212697 30495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:32825 every 8 connection(s)
I20260924 07:18:52.213644 30363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:52.217757 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30363
I20260924 07:18:52.218152 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:39229
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:33357
--webserver_port=39519
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:52.227221 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.215390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:52907 (local address 127.15.127.190:33357)
0924 07:18:52.215530 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.215534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.216412 (+ 878us) server_negotiation.cc:407] Connection header received
0924 07:18:52.217858 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.217862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.217917 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.218023 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:52.219222 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.219750 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.220866 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.221167 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.221279 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.223619 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:52.223623 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:52.223625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:52.223655 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:52.225156 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.225703 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.225707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.225709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.225789 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.226169 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.226173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.226175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.226348 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:52.226449 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.226922 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.227051 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:18:52.227957 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.215836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:52907)
0924 07:18:52.216267 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.216308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.217648 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.218199 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.218205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.218616 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:52.219075 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.219088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.219873 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.219878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.220740 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.220749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.221313 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.221317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.221479 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.222077 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:52.222092 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:52.223450 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:52.225265 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.225270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.225284 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.225574 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.225904 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.225909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.225911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.226070 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.226459 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:52.226467 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:52.226690 (+ 223us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.226870 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.227198 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":106,"threads_started":1}
I20260924 07:18:52.229216 30496 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33357
I20260924 07:18:52.229439 30496 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:52.229907 30496 heartbeater.cc:511] Master 127.15.127.190:33357 requested a full tablet report, sending...
I20260924 07:18:52.230960 30170 ts_manager.cc:194] Registered new tserver with Master: d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
I20260924 07:18:52.231755 30170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:52907
W20260924 07:18:52.328382 30499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:52.328626 30499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:52.328670 30499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:52.332489 30499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:52.332561 30499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:52.332623 30499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:18:52.337597 30499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:39229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=39519
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:52.338572 30499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:52.339542 30499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:52.349056 30504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.349117 30499 server_base.cc:1034] running on GCE node
W20260924 07:18:52.349056 30507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:52.349053 30505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.349690 30499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:52.350260 30499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:52.351426 30499 hybrid_clock.cc:648] HybridClock initialized: now 1790234332351400 us; error 38 us; skew 500 ppm
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:52.354326 30499 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:18:52.355347 30499 webserver.cc:460] Webserver started at http://127.15.127.131:39519/ using document root <none> and password file <none>
I20260924 07:18:52.355964 30499 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:52.356042 30499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:52.360025 30499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260924 07:18:52.362093 30514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.363283 30499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:18:52.363374 30499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "efee6e8cbb844685bba0020a16e76d1c"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "a26e99c36b21686299a7128286af27ff"
server_key_iv: "da34e43338f3f36841129c2b22c6894c"
server_key_version: "encryptionkey@0"
I20260924 07:18:52.363880 30499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:52.379542 30499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:52.382272 30499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:52.382460 30499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:52.383020 30499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:52.383823 30499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:52.383877 30499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.383961 30499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:52.384001 30499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.392693 30499 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:39229
I20260924 07:18:52.392712 30627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:39229 every 8 connection(s)
I20260924 07:18:52.393705 30499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:52.394258 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30499
I20260924 07:18:52.403316 30360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.394939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46766 (local address 127.15.127.190:33357)
0924 07:18:52.395097 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.395102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.395361 (+ 259us) server_negotiation.cc:407] Connection header received
0924 07:18:52.395481 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.395486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.395549 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.395650 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:52.396469 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.397257 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.398593 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.398804 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.398934 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.400313 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:52.400317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:52.400320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:52.400344 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:52.401699 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.402106 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.402111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.402113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.402201 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.402486 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.402489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.402491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.402679 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:52.402884 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.403030 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.403149 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":54}
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:18:52.408003 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.395713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:53503 (local address 127.15.127.190:33357)
0924 07:18:52.395931 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.395936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.396808 (+ 872us) server_negotiation.cc:407] Connection header received
0924 07:18:52.398736 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.398738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.398777 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.398889 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:52.400164 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.400787 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.402033 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.402273 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.402374 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.405092 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:52.405095 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:52.405097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:52.405121 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:52.406338 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.406828 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.406831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.406832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.406899 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:52.407189 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:52.407191 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:52.407192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:52.407330 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:52.407397 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.407810 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.407894 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":155,"thread_start_us":72,"threads_started":1}
I20260924 07:18:52.408746 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.396222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:53503)
0924 07:18:52.396680 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.396718 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.398364 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.399063 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.399070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.399553 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:52.400020 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.400033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.400921 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.400923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.401905 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.401914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.402408 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.402412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.402588 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.403139 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:52.403154 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:52.404913 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:52.406437 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.406441 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.406451 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.406738 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.406971 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:52.406973 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:52.406974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:52.407096 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:52.407408 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:52.407413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:52.407619 (+ 206us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.407754 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.407968 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":134,"threads_started":1}
I20260924 07:18:52.409909 30628 heartbeater.cc:347] Connected to a master server at 127.15.127.190:33357
I20260924 07:18:52.410110 30628 heartbeater.cc:464] Registering TS with master...
I20260924 07:18:52.410547 30628 heartbeater.cc:511] Master 127.15.127.190:33357 requested a full tablet report, sending...
I20260924 07:18:52.411425 30170 ts_manager.cc:194] Registered new tserver with Master: efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229)
I20260924 07:18:52.411922 30170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:53503
I20260924 07:18:52.416791 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:18:52.420619 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.417380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.15.127.190:33357)
0924 07:18:52.417533 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.417536 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.417641 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:18:52.417759 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.417761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.417865 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.417945 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:52.418157 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.418634 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.419411 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.419623 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.419691 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.419931 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:52.420265 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:52.420348 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.420403 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.420449 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 07:18:52.421321 30170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:18:52.424475 30170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:18:52.432621 30292 tablet_service.cc:1467] Processing CreateTablet for tablet ad325415cdd042d3bce2683f9aec1353 (DEFAULT_TABLE table=test-table [id=ed944f21381e4838ace10ad1328410fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:52.433310 30292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad325415cdd042d3bce2683f9aec1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:52.435791 30250 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap starting.
I20260924 07:18:52.436740 30250 tablet_bootstrap.cc:654] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:52.437330 30365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.431167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:39229 (local address 127.0.0.1:44366)
0924 07:18:52.431325 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.431343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.431418 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.432367 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.432370 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.432387 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:52.432484 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.432489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.434323 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.434325 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.435451 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.435462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.436807 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.436810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.436928 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.436970 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.437085 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.437177 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":27}
I20260924 07:18:52.437479 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.431630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:32825 (local address 127.0.0.1:60368)
0924 07:18:52.431985 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.431999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.432099 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.432766 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.432769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.432786 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:52.432875 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.432880 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.434161 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.434165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.435451 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.435462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.437083 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.437086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.437186 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.437205 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.437275 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.437328 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":31,"thread_start_us":121,"threads_started":1}
I20260924 07:18:52.437740 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.431831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60368 (local address 127.15.127.130:32825)
0924 07:18:52.432284 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.432315 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.432338 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:18:52.432409 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.432413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.432621 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.432796 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:52.432955 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.434021 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.435592 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.436590 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.436709 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.436981 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.437333 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.437570 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":146,"threads_started":1}
I20260924 07:18:52.437940 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.431502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44366 (local address 127.15.127.131:39229)
0924 07:18:52.431864 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.431877 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.431900 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:18:52.431977 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.431986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.432199 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.432371 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:52.432608 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.434199 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.435899 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.436683 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.437512 (+ 829us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.437652 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.437704 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.437779 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":134,"threads_started":1}
I20260924 07:18:52.438809 30250 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: No bootstrap required, opened a new log
I20260924 07:18:52.438915 30250 ts_tablet_manager.cc:1397] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:18:52.439299 30250 raft_consensus.cc:348] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.439429 30250 raft_consensus.cc:374] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:52.439610 30250 raft_consensus.cc:729] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e3b77f9054466493eac57ec9aac1d6, State: Initialized, Role: FOLLOWER
I20260924 07:18:52.439957 30250 consensus_queue.cc:260] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.440311 30250 ts_tablet_manager.cc:1428] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260924 07:18:52.440806 30562 tablet_service.cc:1467] Processing CreateTablet for tablet ad325415cdd042d3bce2683f9aec1353 (DEFAULT_TABLE table=test-table [id=ed944f21381e4838ace10ad1328410fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:52.440806 30430 tablet_service.cc:1467] Processing CreateTablet for tablet ad325415cdd042d3bce2683f9aec1353 (DEFAULT_TABLE table=test-table [id=ed944f21381e4838ace10ad1328410fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:18:52.441920 30430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad325415cdd042d3bce2683f9aec1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:52.441929 30562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad325415cdd042d3bce2683f9aec1353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:18:52.449920 30641 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Bootstrap starting.
I20260924 07:18:52.450353 30642 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Bootstrap starting.
I20260924 07:18:52.453037 30642 tablet_bootstrap.cc:654] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:52.453037 30641 tablet_bootstrap.cc:654] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Neither blocks nor log segments found. Creating new log.
I20260924 07:18:52.453972 30641 log.cc:824] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:52.453972 30642 log.cc:824] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:52.455825 30641 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: No bootstrap required, opened a new log
I20260924 07:18:52.455844 30642 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: No bootstrap required, opened a new log
I20260924 07:18:52.456022 30641 ts_tablet_manager.cc:1397] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:18:52.456032 30642 ts_tablet_manager.cc:1397] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:18:52.458815 30641 raft_consensus.cc:348] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.458837 30642 raft_consensus.cc:348] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.459016 30641 raft_consensus.cc:374] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:52.459039 30642 raft_consensus.cc:374] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:18:52.459074 30641 raft_consensus.cc:729] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d97fad6898c943dda8be46b5f5b6e5ae, State: Initialized, Role: FOLLOWER
I20260924 07:18:52.459121 30642 raft_consensus.cc:729] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efee6e8cbb844685bba0020a16e76d1c, State: Initialized, Role: FOLLOWER
I20260924 07:18:52.459527 30641 consensus_queue.cc:260] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.459533 30642 consensus_queue.cc:260] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.460182 30496 heartbeater.cc:503] Master 127.15.127.190:33357 was elected leader, sending a full tablet report...
I20260924 07:18:52.460367 30628 heartbeater.cc:503] Master 127.15.127.190:33357 was elected leader, sending a full tablet report...
I20260924 07:18:52.460528 30641 ts_tablet_manager.cc:1428] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260924 07:18:52.460645 30642 ts_tablet_manager.cc:1428] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260924 07:18:52.464457 30497 tablet.cc:2369] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:52.519101 30359 tablet.cc:2369] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:18:52.644677 30629 tablet.cc:2369] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:18:52.654651 30645 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:52.654861 30645 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.656121 30645 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825), 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899)
I20260924 07:18:52.660176 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.656420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:32825 (local address 127.15.127.131:34309)
0924 07:18:52.656575 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.656594 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.656696 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.657099 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.657102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.657119 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:52.657252 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.657259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.658078 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.658081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.658987 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.658997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.659725 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.659729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.659845 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.659870 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.659944 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.660017 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":49}
I20260924 07:18:52.660187 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.656683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:34309 (local address 127.15.127.130:32825)
0924 07:18:52.656833 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.656838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.656855 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:18:52.656913 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.656916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.656981 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.657084 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:52.657381 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.657970 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.659115 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.659608 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.659703 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.659743 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.659974 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.660061 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:18:52.660351 30647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.656555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:56657 (local address 127.15.127.129:34899)
0924 07:18:52.656937 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.656941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.657038 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:18:52.657208 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.657213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.657279 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.657396 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:18:52.657639 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.658160 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.659242 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.659747 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.659864 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.660014 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.660093 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.660164 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":99,"threads_started":1}
I20260924 07:18:52.660746 30648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.656420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34899 (local address 127.15.127.131:56657)
0924 07:18:52.656918 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.656930 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:52.657095 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:52.657405 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:52.657408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:52.657424 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:18:52.657532 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.657538 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.658281 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.658283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.659032 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:52.659041 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.659862 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.659864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:52.659969 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.659993 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:18:52.660075 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:18:52.660141 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":155,"thread_start_us":111,"threads_started":1}
I20260924 07:18:52.660948 30450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad325415cdd042d3bce2683f9aec1353" candidate_uuid: "efee6e8cbb844685bba0020a16e76d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" is_pre_election: true
I20260924 07:18:52.661293 30450 raft_consensus.cc:2454] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efee6e8cbb844685bba0020a16e76d1c in term 0.
I20260924 07:18:52.661309 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad325415cdd042d3bce2683f9aec1353" candidate_uuid: "efee6e8cbb844685bba0020a16e76d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e3b77f9054466493eac57ec9aac1d6" is_pre_election: true
I20260924 07:18:52.661597 30312 raft_consensus.cc:2454] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efee6e8cbb844685bba0020a16e76d1c in term 0.
I20260924 07:18:52.661788 30515 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c; no voters:
I20260924 07:18:52.662081 30645 raft_consensus.cc:2781] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:18:52.662155 30645 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:18:52.662185 30645 raft_consensus.cc:3037] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:52.663441 30645 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.663807 30645 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [CANDIDATE]: Term 1 election: Requested vote from peers d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825), 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899)
I20260924 07:18:52.664094 30450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad325415cdd042d3bce2683f9aec1353" candidate_uuid: "efee6e8cbb844685bba0020a16e76d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d97fad6898c943dda8be46b5f5b6e5ae"
I20260924 07:18:52.664165 30312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad325415cdd042d3bce2683f9aec1353" candidate_uuid: "efee6e8cbb844685bba0020a16e76d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08e3b77f9054466493eac57ec9aac1d6"
I20260924 07:18:52.664233 30450 raft_consensus.cc:3037] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:52.664300 30312 raft_consensus.cc:3037] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:18:52.665302 30312 raft_consensus.cc:2454] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efee6e8cbb844685bba0020a16e76d1c in term 1.
I20260924 07:18:52.665345 30450 raft_consensus.cc:2454] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efee6e8cbb844685bba0020a16e76d1c in term 1.
I20260924 07:18:52.665576 30517 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6, efee6e8cbb844685bba0020a16e76d1c; no voters:
I20260924 07:18:52.665793 30645 raft_consensus.cc:2781] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:18:52.666011 30645 raft_consensus.cc:686] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 1 LEADER]: Becoming Leader. State: Replica: efee6e8cbb844685bba0020a16e76d1c, State: Running, Role: LEADER
I20260924 07:18:52.666339 30645 consensus_queue.cc:237] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:52.669426 30170 catalog_manager.cc:5495] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c reported cstate change: term changed from 0 to 1, leader changed from <none> to efee6e8cbb844685bba0020a16e76d1c (127.15.127.131). New cstate: current_term: 1 leader_uuid: "efee6e8cbb844685bba0020a16e76d1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } health_report { overall_health: UNKNOWN } } }
I20260924 07:18:52.728364 30637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.725234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44370 (local address 127.15.127.131:39229)
0924 07:18:52.725372 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:52.725376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:52.725388 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:18:52.725462 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:52.725464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:52.725519 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:52.725617 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:18:52.725831 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.726313 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.727076 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:52.727278 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:52.727350 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:52.727607 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:18:52.728018 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:18:52.728108 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:52.728160 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 07:18:52.728208 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:18:52.734130 30450 raft_consensus.cc:1261] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Refusing update from remote peer efee6e8cbb844685bba0020a16e76d1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:52.734127 30312 raft_consensus.cc:1261] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Refusing update from remote peer efee6e8cbb844685bba0020a16e76d1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:18:52.734823 30649 consensus_queue.cc:1035] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:52.735069 30645 consensus_queue.cc:1035] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:18:52.740245 30652 mvcc.cc:205] Tried to move back new op lower bound from 7332799826871693312 to 7332799826603999232. Current Snapshot: MvccSnapshot[applied={T|T < 7332799826871693312}]
I20260924 07:18:52.742619 30654 mvcc.cc:205] Tried to move back new op lower bound from 7332799826871693312 to 7332799826603999232. Current Snapshot: MvccSnapshot[applied={T|T < 7332799826871693312}]
I20260924 07:18:52.742794 30653 mvcc.cc:205] Tried to move back new op lower bound from 7332799826871693312 to 7332799826603999232. Current Snapshot: MvccSnapshot[applied={T|T < 7332799826871693312}]
I20260924 07:18:52.748507 30169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:52.748687 30169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:18:52.750946 30169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Sending DeleteTablet for 3 replicas of tablet ad325415cdd042d3bce2683f9aec1353
I20260924 07:18:52.751523 30562 tablet_service.cc:1514] Processing DeleteTablet for tablet ad325415cdd042d3bce2683f9aec1353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:52 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:44366
I20260924 07:18:52.751574 30430 tablet_service.cc:1514] Processing DeleteTablet for tablet ad325415cdd042d3bce2683f9aec1353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:52 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:60368
I20260924 07:18:52.751668 30292 tablet_service.cc:1514] Processing DeleteTablet for tablet ad325415cdd042d3bce2683f9aec1353 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:18:52 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:34768
I20260924 07:18:52.751968 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 30208
I20260924 07:18:52.752082 30663 tablet_replica.cc:330] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: stopping tablet replica
I20260924 07:18:52.752369 30663 raft_consensus.cc:2229] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:18:52.752708 30663 raft_consensus.cc:2258] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:52.753077 30662 tablet_replica.cc:330] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: stopping tablet replica
I20260924 07:18:52.753376 30662 raft_consensus.cc:2229] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:18:52.753679 30662 raft_consensus.cc:2258] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:18:52.754258 30663 ts_tablet_manager.cc:1905] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:52.755008 30662 ts_tablet_manager.cc:1905] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:18:52.756939 30663 ts_tablet_manager.cc:1918] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:52.757102 30663 log.cc:1196] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/ad325415cdd042d3bce2683f9aec1353
I20260924 07:18:52.757434 30663 ts_tablet_manager.cc:1939] T ad325415cdd042d3bce2683f9aec1353 P efee6e8cbb844685bba0020a16e76d1c: Deleting consensus metadata
I20260924 07:18:52.757663 30662 ts_tablet_manager.cc:1918] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:18:52.757754 30662 log.cc:1196] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/ad325415cdd042d3bce2683f9aec1353
I20260924 07:18:52.758097 30662 ts_tablet_manager.cc:1939] T ad325415cdd042d3bce2683f9aec1353 P d97fad6898c943dda8be46b5f5b6e5ae: Deleting consensus metadata
I20260924 07:18:52.758646 30157 catalog_manager.cc:4841] TS efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): tablet ad325415cdd042d3bce2683f9aec1353 (table test-table [id=ed944f21381e4838ace10ad1328410fb]) successfully deleted
I20260924 07:18:52.759151 30154 catalog_manager.cc:4841] TS d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): tablet ad325415cdd042d3bce2683f9aec1353 (table test-table [id=ed944f21381e4838ace10ad1328410fb]) successfully deleted
W20260924 07:18:52.765205 30156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:34899 (error 108)
I20260924 07:18:52.765465 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 30363
I20260924 07:18:52.767096 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:52.766756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:34899 (local address 127.0.0.1:34770)
0924 07:18:52.766951 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:52.767012 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:34899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260924 07:18:52.767382 30156 catalog_manager.cc:4584] TS 08e3b77f9054466493eac57ec9aac1d6 (127.15.127.129:34899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ad325415cdd042d3bce2683f9aec1353: Network error: Client connection negotiation failed: client connection to 127.15.127.129:34899: connect: Connection refused (error 111)
I20260924 07:18:52.771749 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 30499
I20260924 07:18:52.777619 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 30137
I20260924 07:18:52.783828 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33357
--webserver_interface=127.15.127.190
--webserver_port=37629
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:18:52.895370 30666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:52.895633 30666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:52.895715 30666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:52.899838 30666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:18:52.899890 30666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:52.899909 30666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:52.899962 30666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:18:52.899991 30666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:18:52.905377 30666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=37629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:52.906376 30666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:52.907377 30666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:52.915709 30672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.915828 30666 server_base.cc:1034] running on GCE node
W20260924 07:18:52.915750 30671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:52.915731 30674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:52.916584 30666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:52.917704 30666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:52.918860 30666 hybrid_clock.cc:648] HybridClock initialized: now 1790234332918825 us; error 50 us; skew 500 ppm
Sep 24 07:18:52 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234332, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:52.921540 30666 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:18:52.922448 30666 webserver.cc:460] Webserver started at http://127.15.127.190:37629/ using document root <none> and password file <none>
I20260924 07:18:52.923012 30666 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:52.923080 30666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:52.926817 30666 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 07:18:52.928838 30681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:52.930173 30666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:18:52.930313 30666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "2114a796eeb64057be8ded29eca472de"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "50370396b5d7e95f6f12f79130106830"
server_key_iv: "06c5248f3ff9d18a29f66262fc341743"
server_key_version: "encryptionkey@0"
I20260924 07:18:52.930851 30666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:52.967633 30666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:52.970564 30666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:52.970794 30666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:52.977701 30666 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33357
I20260924 07:18:52.977985 30733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33357 every 8 connection(s)
I20260924 07:18:52.978798 30666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:18:52.980780 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30666
I20260924 07:18:52.981226 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:34899
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:33357
--webserver_port=39999
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:52.983201 30734 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:18:52.984814 30734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap starting.
I20260924 07:18:53.009385 30734 log.cc:824] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:53.021533 30734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:53.021927 30734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de: Bootstrap complete.
I20260924 07:18:53.026021 30734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:53.026417 30734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2114a796eeb64057be8ded29eca472de, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.026898 30734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:53.027011 30734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:18:53.027103 30734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:18:53.027220 30734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:18:53.029116 30734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:53.029453 30734 leader_election.cc:302] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2114a796eeb64057be8ded29eca472de; no voters:
I20260924 07:18:53.029775 30734 leader_election.cc:288] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:18:53.029855 30738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:18:53.030037 30738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [term 3 LEADER]: Becoming Leader. State: Replica: 2114a796eeb64057be8ded29eca472de, State: Running, Role: LEADER
I20260924 07:18:53.030315 30738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } }
I20260924 07:18:53.030961 30734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:18:53.031222 30738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2114a796eeb64057be8ded29eca472de. Latest consensus state: current_term: 3 leader_uuid: "2114a796eeb64057be8ded29eca472de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } } }
I20260924 07:18:53.031337 30738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:53.031689 30742 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:18:53.031719 30739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2114a796eeb64057be8ded29eca472de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2114a796eeb64057be8ded29eca472de" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33357 } } }
I20260924 07:18:53.031931 30739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2114a796eeb64057be8ded29eca472de [sys.catalog]: This master's current role is: LEADER
I20260924 07:18:53.036664 30742 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:18:53.037477 30742 catalog_manager.cc:1259] Loaded cluster ID: 5c1fe7223fd84fcfb0dbdc5c27f424e6
I20260924 07:18:53.037536 30742 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 07:18:53.038555 30742 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe4fa89a9ad google::LogMessage::Fail() at ??:0
@ 0x7fe4fa89d88a google::LogMessage::SendToLog() at ??:0
@ 0x7fe4fa89a4de google::LogMessage::Flush() at ??:0
@ 0x7fe4fa89ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5051d9fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe5051dad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe5051d7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe50521b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4fcaec3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4fcaecc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe4fcaee3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe4fcadd512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4fe1216db start_thread at ??:0
@ 0x7fe4f939861f clone at ??:0
W20260924 07:18:53.133288 30736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:53.133585 30736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:53.133666 30736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:53.137713 30736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:53.137790 30736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:53.137877 30736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:18:53.143520 30736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:34899
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=39999
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:53.144622 30736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:53.145699 30736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:53.155361 30776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.155370 30778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.155361 30775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.155709 30736 server_base.cc:1034] running on GCE node
I20260924 07:18:53.156140 30736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:53.156814 30736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:53.158000 30736 hybrid_clock.cc:648] HybridClock initialized: now 1790234333157982 us; error 42 us; skew 500 ppm
Sep 24 07:18:53 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:53.166203 30736 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:18:53.167183 30736 webserver.cc:460] Webserver started at http://127.15.127.129:39999/ using document root <none> and password file <none>
I20260924 07:18:53.167768 30736 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:53.167830 30736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:53.171893 30736 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:18:53.174198 30785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.175523 30736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:53.175668 30736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "08e3b77f9054466493eac57ec9aac1d6"
format_stamp: "Formatted at 2026-09-24 07:18:50 on dist-test-slave-zx46"
server_key: "c39522d68f8f489e55771a281109523a"
server_key_iv: "37b2e46fe832091d7a773039d3815048"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.176221 30736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:53.217859 30736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:53.221273 30736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:53.221474 30736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:53.222069 30736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:53.223196 30792 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:18:53.228595 30736 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:18:53.228665 30736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 07:18:53.228734 30736 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:18:53.230464 30736 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:18:53.230520 30736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 07:18:53.230767 30792 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap starting.
I20260924 07:18:53.241412 30736 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:34899
I20260924 07:18:53.241644 30899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:34899 every 8 connection(s)
I20260924 07:18:53.242668 30736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:18:53.250765 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30736
I20260924 07:18:53.251233 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:32825
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:33357
--webserver_port=46817
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:53.251775 30792 log.cc:824] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:18:53.260944 30792 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:18:53.261395 30792 tablet_bootstrap.cc:492] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Bootstrap complete.
I20260924 07:18:53.262681 30792 ts_tablet_manager.cc:1397] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Time spent bootstrapping tablet: real 0.032s user 0.024s sys 0.006s
W20260924 07:18:53.263279 30902 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:18:53.245079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:39329)
0924 07:18:53.245679 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.245749 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:53.246866 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:53.262167 (+ 15301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":127,"threads_started":1}
W20260924 07:18:53.263844 30900 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:33357 (0 consecutive failures): Network error: Failed to ping master at 127.15.127.190:33357: Client connection negotiation failed: client connection to 127.15.127.190:33357: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:18:53.264545 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.264270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:34953)
0924 07:18:53.264407 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.264478 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:18:53.265347 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.265120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:39269)
0924 07:18:53.265234 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.265286 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:18:53.265580 30900 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:18:53.265829 30792 raft_consensus.cc:348] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:53.266212 30792 raft_consensus.cc:729] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e3b77f9054466493eac57ec9aac1d6, State: Initialized, Role: FOLLOWER
I20260924 07:18:53.266665 30792 consensus_queue.cc:260] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:53.267611 30792 ts_tablet_manager.cc:1428] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 07:18:53.267693 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.267479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:47583)
0924 07:18:53.267585 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.267641 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:18:53.268304 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.268098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:49573)
0924 07:18:53.268201 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.268248 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:18:53.362295 30904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:53.362525 30904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:53.362572 30904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:53.366529 30904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:53.366583 30904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:53.366639 30904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:18:53.371681 30904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:32825
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=46817
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:53.372632 30904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:53.373647 30904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:53.382898 30913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.382918 30911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.382915 30910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.383139 30904 server_base.cc:1034] running on GCE node
I20260924 07:18:53.383494 30904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:53.384121 30904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:53.385299 30904 hybrid_clock.cc:648] HybridClock initialized: now 1790234333385289 us; error 30 us; skew 500 ppm
Sep 24 07:18:53 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:53.387890 30904 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:18:53.388645 30904 webserver.cc:460] Webserver started at http://127.15.127.130:46817/ using document root <none> and password file <none>
I20260924 07:18:53.389209 30904 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:53.389271 30904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:53.392839 30904 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:18:53.394861 30920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.396010 30904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:18:53.396111 30904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "d97fad6898c943dda8be46b5f5b6e5ae"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "375ba514145a57dcd9298a8ae5205342"
server_key_iv: "ddb089eb6cd106a87669a8580e3a841f"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.396595 30904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:53.415201 30904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:53.417961 30904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:53.418115 30904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:53.418617 30904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:53.419472 30904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:53.419520 30904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.419559 30904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:53.419575 30904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.428305 30904 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:32825
I20260924 07:18:53.428342 31033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:32825 every 8 connection(s)
I20260924 07:18:53.429361 30904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:18:53.430102 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 30904
I20260924 07:18:53.430629 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:39229
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:33357
--webserver_port=39519
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:44605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:18:53.433905 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.431770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:35533)
0924 07:18:53.432364 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.432965 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":146,"threads_started":1}
W20260924 07:18:53.434357 31034 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:33357 (0 consecutive failures): Network error: Failed to ping master at 127.15.127.190:33357: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
I20260924 07:18:53.434986 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.434777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:41115)
0924 07:18:53.434888 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.434934 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:18:53.435693 31037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.435401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:40201)
0924 07:18:53.435536 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.435611 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 07:18:53.435897 31034 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:18:53.541708 31036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:18:53.541963 31036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:18:53.542007 31036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:18:53.546006 31036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:18:53.546068 31036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:18:53.546161 31036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:18:53.551350 31036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:44605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:39229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=39519
--tserver_master_addrs=127.15.127.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:18:53.552425 31036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:18:53.553516 31036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:18:53.563274 31042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.563237 31045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:18:53.563328 31043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:18:53.563495 31036 server_base.cc:1034] running on GCE node
I20260924 07:18:53.563979 31036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:18:53.564630 31036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:18:53.565812 31036 hybrid_clock.cc:648] HybridClock initialized: now 1790234333565797 us; error 28 us; skew 500 ppm
Sep 24 07:18:53 dist-test-slave-zx46 krb5kdc[29590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:18:53.568647 31036 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:18:53.569609 31036 webserver.cc:460] Webserver started at http://127.15.127.131:39519/ using document root <none> and password file <none>
I20260924 07:18:53.570132 31036 fs_manager.cc:359] Metadata directory not provided
I20260924 07:18:53.570194 31036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:18:53.574085 31036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:18:53.576206 31052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.577596 31036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:18:53.577692 31036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "efee6e8cbb844685bba0020a16e76d1c"
format_stamp: "Formatted at 2026-09-24 07:18:51 on dist-test-slave-zx46"
server_key: "a26e99c36b21686299a7128286af27ff"
server_key_iv: "da34e43338f3f36841129c2b22c6894c"
server_key_version: "encryptionkey@0"
I20260924 07:18:53.578217 31036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:18:53.645839 31036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:18:53.648772 31036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:18:53.648940 31036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:18:53.649503 31036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:18:53.650348 31036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:18:53.650422 31036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.650496 31036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:18:53.650540 31036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:18:53.660107 31036 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:39229
I20260924 07:18:53.660131 31165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:39229 every 8 connection(s)
I20260924 07:18:53.661131 31036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:18:53.664885 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.663308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:53413)
0924 07:18:53.663765 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.664177 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":114,"threads_started":1}
W20260924 07:18:53.665391 31166 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:33357 (0 consecutive failures): Network error: Failed to ping master at 127.15.127.190:33357: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
I20260924 07:18:53.665927 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.665733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:43657)
0924 07:18:53.665829 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.665875 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:18:53.666548 31168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:53.666380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:58579)
0924 07:18:53.666464 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:53.666503 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:18:53.666776 31166 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:18:53.668463 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31036
I20260924 07:18:54.269603 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.269125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:56107)
0924 07:18:54.269390 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.269442 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260924 07:18:54.436779 31171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.436419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:56493)
0924 07:18:54.436637 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.436687 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
I20260924 07:18:54.667742 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.667348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:35415)
0924 07:18:54.667589 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.667646 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260924 07:18:54.782799 31173 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:54.783159 31173 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:54.784271 31173 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
Sep 24 07:18:54 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.131@KRBTEST.COM
Sep 24 07:18:54 dist-test-slave-zx46 krb5kdc[29590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234333, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.130@KRBTEST.COM
I20260924 07:18:54.798652 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.784504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:32825 (local address 127.15.127.129:51431)
0924 07:18:54.784817 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.784845 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.784963 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.787014 (+ 2051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.787019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.787441 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:54.788064 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.788078 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.789361 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.789369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.790220 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.790228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.790898 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.790902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.791098 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.791803 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:54.791814 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:54.793755 (+ 1941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:54.795942 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:54.795948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:54.795951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:54.796287 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:54.796646 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:54.796649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:54.796651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:54.796849 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:54.797575 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:54.797579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:54.797945 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.798189 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.798478 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":116,"spinlock_wait_cycles":39168,"thread_start_us":69,"threads_started":1}
I20260924 07:18:54.798938 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.784669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:51431 (local address 127.15.127.130:32825)
0924 07:18:54.785167 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.785184 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.785217 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:18:54.786132 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.786141 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.786612 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.786946 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:54.788250 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.789226 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.790345 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.790777 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.790954 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.793955 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:54.793964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:54.793982 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:54.794026 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:54.795852 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:54.796394 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:54.796399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:54.796404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:54.796564 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:54.796938 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:54.796941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:54.796942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:54.797455 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:54.797594 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.798477 (+ 883us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.798747 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":120,"threads_started":1}
I20260924 07:18:54.799077 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.784504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:39229 (local address 127.15.127.129:33021)
0924 07:18:54.785256 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:54.785267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:18:54.785348 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:18:54.787548 (+ 2200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:18:54.787552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:18:54.787772 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:18:54.788059 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.788075 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.789362 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.789366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.790222 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:18:54.790231 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.790904 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.790908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:18:54.791093 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.791746 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:18:54.791762 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:18:54.793407 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:18:54.795844 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:54.795849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:54.795858 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:54.796287 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:54.796646 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:18:54.796649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:18:54.796651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:18:54.796844 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:18:54.797573 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:18:54.797579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:18:54.797945 (+ 366us) client_negotiation.cc:769] Sending connection context
0924 07:18:54.798182 (+ 237us) client_negotiation.cc:240] Negotiation successful
0924 07:18:54.798883 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":676,"spinlock_wait_cycles":11904,"thread_start_us":112,"threads_started":1}
I20260924 07:18:54.799074 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:54.784666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:33021 (local address 127.15.127.131:39229)
0924 07:18:54.784975 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 07:18:54.785002 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 07:18:54.785360 (+ 358us) server_negotiation.cc:407] Connection header received
0924 07:18:54.786132 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:18:54.786141 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:18:54.786612 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:18:54.786945 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:18:54.788229 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.789233 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.790356 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:18:54.790784 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:18:54.790980 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:18:54.793639 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:18:54.793645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:18:54.793656 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:18:54.793683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:18:54.795730 (+ 2047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:54.796394 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:54.796399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:54.796404 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:54.796550 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:18:54.796938 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:18:54.796941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:18:54.796942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:18:54.797454 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:18:54.797589 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:18:54.798317 (+ 728us) server_negotiation.cc:299] Negotiation successful
0924 07:18:54.798597 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
W20260924 07:18:54.801508 30789 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
W20260924 07:18:54.801985 30786 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
I20260924 07:18:54.802127 30786 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6; no voters: d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c
I20260924 07:18:54.802402 31173 raft_consensus.cc:2726] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:18:55.270735 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:55.270423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:58019)
0924 07:18:55.270578 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:55.270650 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 07:18:55.437835 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:55.437452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:34257)
0924 07:18:55.437694 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:55.437748 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":90,"threads_started":1}
I20260924 07:18:55.668836 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:55.668454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:35845)
0924 07:18:55.668696 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:55.668749 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
I20260924 07:18:56.271912 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:56.271480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:43331)
0924 07:18:56.271762 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:56.271818 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260924 07:18:56.439041 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:56.438577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:32891)
0924 07:18:56.438837 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:56.438935 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":72,"threads_started":1}
I20260924 07:18:56.669832 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:56.669480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:35313)
0924 07:18:56.669690 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:56.669739 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":67,"threads_started":1}
I20260924 07:18:56.836064 31183 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:56.836194 31183 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:56.836673 31183 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
W20260924 07:18:56.837477 30789 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
W20260924 07:18:56.837589 30786 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
I20260924 07:18:56.837682 30786 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6; no voters: d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c
I20260924 07:18:56.837826 31183 raft_consensus.cc:2726] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:18:57.273185 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.272706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:37825)
0924 07:18:57.272958 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:57.273077 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260924 07:18:57.440189 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.439797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:33261)
0924 07:18:57.440025 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:57.440086 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260924 07:18:57.670981 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:57.670520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:42313)
0924 07:18:57.670838 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:57.670897 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":74,"threads_started":1}
I20260924 07:18:58.274430 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.274004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:55195)
0924 07:18:58.274275 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.274333 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
I20260924 07:18:58.376242 31188 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:18:58.376384 31188 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:18:58.376806 31188 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
W20260924 07:18:58.377560 30789 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
W20260924 07:18:58.377736 30786 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
I20260924 07:18:58.377830 30786 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6; no voters: d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c
I20260924 07:18:58.377978 31188 raft_consensus.cc:2726] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:18:58.441524 31189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.441160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:34969)
0924 07:18:58.441386 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.441439 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260924 07:18:58.672086 31190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:58.671686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:59877)
0924 07:18:58.671932 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:58.671985 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":83,"threads_started":1}
I20260924 07:18:59.275517 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:59.275123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:35825)
0924 07:18:59.275375 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:59.275426 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":71,"threads_started":1}
I20260924 07:18:59.442806 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:59.442257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:56353)
0924 07:18:59.442503 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:59.442558 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260924 07:18:59.673256 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:18:59.672803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:43167)
0924 07:18:59.673098 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 07:18:59.673161 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260924 07:19:00.276700 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:00.276265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:55327)
0924 07:19:00.276531 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:00.276605 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":91,"threads_started":1}
I20260924 07:19:00.443917 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:00.443475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:49589)
0924 07:19:00.443767 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:00.443822 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260924 07:19:00.674394 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:00.673953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:58497)
0924 07:19:00.674243 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:00.674301 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":73,"threads_started":1}
I20260924 07:19:01.277912 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:01.277505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:36623)
0924 07:19:01.277764 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:01.277823 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260924 07:19:01.445176 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:01.444687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:52225)
0924 07:19:01.444964 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:01.445080 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":70,"threads_started":1}
I20260924 07:19:01.675503 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:01.675126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:40665)
0924 07:19:01.675362 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:01.675413 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":83,"threads_started":1}
I20260924 07:19:02.279064 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:02.278599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:60801)
0924 07:19:02.278864 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:02.278956 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260924 07:19:02.446308 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:02.445872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:47177)
0924 07:19:02.446145 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:02.446204 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260924 07:19:02.676687 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:02.676238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:57029)
0924 07:19:02.676515 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:02.676583 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":100,"threads_started":1}
I20260924 07:19:02.706596 31203 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:02.706730 31203 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:19:02.707247 31203 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
W20260924 07:19:02.708184 30786 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
W20260924 07:19:02.708300 30789 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
I20260924 07:19:02.708426 30789 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6; no voters: d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c
I20260924 07:19:02.708662 31203 raft_consensus.cc:2726] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:19:03.280419 31204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:03.279868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:45523)
0924 07:19:03.280228 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:03.280296 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":108,"threads_started":1}
I20260924 07:19:03.447518 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:03.447098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:56597)
0924 07:19:03.447344 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:03.447401 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260924 07:19:03.677933 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:03.677482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:44475)
0924 07:19:03.677786 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:03.677841 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":143,"threads_started":1}
I20260924 07:19:04.274412 31207 raft_consensus.cc:482] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:04.274544 31207 raft_consensus.cc:504] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efee6e8cbb844685bba0020a16e76d1c" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 39229 } } peers { permanent_uuid: "d97fad6898c943dda8be46b5f5b6e5ae" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 32825 } } peers { permanent_uuid: "08e3b77f9054466493eac57ec9aac1d6" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 34899 } }
I20260924 07:19:04.274915 31207 leader_election.cc:288] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229), d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825)
W20260924 07:19:04.275713 30789 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer efee6e8cbb844685bba0020a16e76d1c (127.15.127.131:39229): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
W20260924 07:19:04.275810 30786 leader_election.cc:341] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d97fad6898c943dda8be46b5f5b6e5ae (127.15.127.130:32825): Not found: Tablet not found: ad325415cdd042d3bce2683f9aec1353
I20260924 07:19:04.275916 30786 leader_election.cc:302] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08e3b77f9054466493eac57ec9aac1d6; no voters: d97fad6898c943dda8be46b5f5b6e5ae, efee6e8cbb844685bba0020a16e76d1c
I20260924 07:19:04.276094 31207 raft_consensus.cc:2726] T ad325415cdd042d3bce2683f9aec1353 P 08e3b77f9054466493eac57ec9aac1d6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:19:04.281445 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:04.281109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:56537)
0924 07:19:04.281311 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:04.281365 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":64,"threads_started":1}
I20260924 07:19:04.448930 31209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:04.448363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:38667)
0924 07:19:04.448754 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:04.448833 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":71,"threads_started":1}
I20260924 07:19:04.679109 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:04.678685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:41485)
0924 07:19:04.678962 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:04.679012 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260924 07:19:05.282682 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:05.282241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:43349)
0924 07:19:05.282500 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:05.282588 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260924 07:19:05.450062 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:05.449663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:36021)
0924 07:19:05.449915 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:05.449968 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260924 07:19:05.680320 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:05.679853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:33463)
0924 07:19:05.680131 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:05.680210 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":106,"threads_started":1}
I20260924 07:19:06.284011 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:06.283530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:42465)
0924 07:19:06.283843 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:06.283911 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260924 07:19:06.451339 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:06.450868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:53891)
0924 07:19:06.451174 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:06.451242 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
I20260924 07:19:06.681576 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:06.681150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:51087)
0924 07:19:06.681411 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:06.681478 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":74,"threads_started":1}
I20260924 07:19:07.285307 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:07.284823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:54427)
0924 07:19:07.285142 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:07.285210 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":74,"threads_started":1}
I20260924 07:19:07.452672 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:07.452163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:44183)
0924 07:19:07.452484 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:07.452555 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":95,"threads_started":1}
I20260924 07:19:07.682829 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:07.682381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.131:56023)
0924 07:19:07.682670 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:07.682733 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":92,"threads_started":1}
I20260924 07:19:08.286656 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:08.286239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.129:42181)
0924 07:19:08.286493 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:08.286551 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260924 07:19:08.454041 31221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:08.453611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33357 (local address 127.15.127.130:60219)
0924 07:19:08.453876 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:08.453938 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:33357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260924 07:19:08.670692 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 30736
I20260924 07:19:08.676445 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 30904
I20260924 07:19:08.683619 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31036
2026-09-24T07:19:08Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19306 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:19:08 dist-test-slave-zx46 krb5kdc[31224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:19:08 dist-test-slave-zx46 krb5kdc[31224](info): set up 2 sockets
Sep 24 07:19:08 dist-test-slave-zx46 krb5kdc[31224](info): commencing operation
krb5kdc: starting...
W20260924 07:19:09.717375 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:19:09 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:19:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:19:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:19:09.836892 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:37131
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:09.945796 31237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:09.946046 31237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:09.946085 31237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:09.949968 31237 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:19:09.950017 31237 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:19:09.950033 31237 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:19:09.950047 31237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:09.950065 31237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:09.950083 31237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:09.950099 31237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:09.950114 31237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:09.955499 31237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:09.956480 31237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:09.957533 31237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:09.965914 31242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:09.966028 31237 server_base.cc:1034] running on GCE node
W20260924 07:19:09.965919 31243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:09.966223 31245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:09.966815 31237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:09.967799 31237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:09.968982 31237 hybrid_clock.cc:648] HybridClock initialized: now 1790234349968951 us; error 41 us; skew 500 ppm
Sep 24 07:19:09 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234349, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:09.971765 31237 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:19:09.972606 31237 webserver.cc:460] Webserver started at http://127.15.127.190:36035/ using document root <none> and password file <none>
I20260924 07:19:09.973184 31237 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:09.973253 31237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:09.973419 31237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:09.975381 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "72810508f1ef4581a529f869a1098fbb"
format_stamp: "Formatted at 2026-09-24 07:19:09 on dist-test-slave-zx46"
server_key: "694394066c374ced8b770f9080074384"
server_key_iv: "577747b9e01c896543a3419768203247"
server_key_version: "encryptionkey@0"
I20260924 07:19:09.975896 31237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "72810508f1ef4581a529f869a1098fbb"
format_stamp: "Formatted at 2026-09-24 07:19:09 on dist-test-slave-zx46"
server_key: "694394066c374ced8b770f9080074384"
server_key_iv: "577747b9e01c896543a3419768203247"
server_key_version: "encryptionkey@0"
I20260924 07:19:09.979547 31237 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 07:19:09.981954 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:09.983178 31237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:19:09.983325 31237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "72810508f1ef4581a529f869a1098fbb"
format_stamp: "Formatted at 2026-09-24 07:19:09 on dist-test-slave-zx46"
server_key: "694394066c374ced8b770f9080074384"
server_key_iv: "577747b9e01c896543a3419768203247"
server_key_version: "encryptionkey@0"
I20260924 07:19:09.983436 31237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:10.000613 31237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:10.004310 31237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:10.004551 31237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:10.011674 31237 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:37131
I20260924 07:19:10.011993 31304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:37131 every 8 connection(s)
I20260924 07:19:10.012671 31237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:10.015794 31305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:10.022936 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31237
I20260924 07:19:10.023015 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap starting.
I20260924 07:19:10.023059 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:19:10.023365 15870 external_mini_cluster.cc:1428] Setting key 4369be2c461d66c7a15d25baaa2d69ae
I20260924 07:19:10.025650 31305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:10.026417 31305 log.cc:824] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:10.028255 31305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: No bootstrap required, opened a new log
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:10.032617 31305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:10.032858 31305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:10.032936 31305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72810508f1ef4581a529f869a1098fbb, State: Initialized, Role: FOLLOWER
I20260924 07:19:10.033470 31305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:10.033663 31305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:10.033739 31305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:10.033841 31305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:10.034734 31305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:10.035092 31305 leader_election.cc:302] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72810508f1ef4581a529f869a1098fbb; no voters:
I20260924 07:19:10.035439 31305 leader_election.cc:288] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:19:10.035511 31310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:10.035735 31310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 LEADER]: Becoming Leader. State: Replica: 72810508f1ef4581a529f869a1098fbb, State: Running, Role: LEADER
I20260924 07:19:10.036082 31310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:10.036715 31305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:10.037379 31312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72810508f1ef4581a529f869a1098fbb. Latest consensus state: current_term: 1 leader_uuid: "72810508f1ef4581a529f869a1098fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } } }
I20260924 07:19:10.037599 31312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:10.037897 31311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72810508f1ef4581a529f869a1098fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } } }
I20260924 07:19:10.037999 31311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:10.038048 31319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:10.038100 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.024638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.15.127.190:37131)
0924 07:19:10.025049 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.025068 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.025103 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:19:10.025767 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.025791 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.026125 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.026390 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.027265 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.028225 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.029406 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.029874 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.029990 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.032796 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:10.032801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:10.032814 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:10.032840 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:10.034786 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.035223 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.035229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.035235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.035351 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.035659 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.035663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.035664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.036118 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:10.036334 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.036595 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.037072 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":129,"threads_started":1}
I20260924 07:19:10.041540 31319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:10.047281 31319 catalog_manager.cc:1347] Generated new cluster ID: 791312e0b861401cb76309251784bbc1
I20260924 07:19:10.047369 31319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:19:10.060304 31319 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:19:10.061543 31319 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:10.071179 31319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Generated new TSK 0
I20260924 07:19:10.071997 31319 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:19:10.101166 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:37131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:10.214481 31331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:10.214735 31331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:10.214778 31331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:10.218922 31331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:10.218981 31331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:10.219094 31331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:19:10.224296 31331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:10.225412 31331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:10.226488 31331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:10.235826 31337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:10.235826 31339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:10.235837 31336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:10.235999 31331 server_base.cc:1034] running on GCE node
I20260924 07:19:10.236634 31331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:10.237325 31331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:10.238474 31331 hybrid_clock.cc:648] HybridClock initialized: now 1790234350238456 us; error 29 us; skew 500 ppm
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234350, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:10.241200 31331 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:19:10.241981 31331 webserver.cc:460] Webserver started at http://127.15.127.129:34811/ using document root <none> and password file <none>
I20260924 07:19:10.242476 31331 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:10.242534 31331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:10.242700 31331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:10.244654 31331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "0effc2046fa3492da9ef20dd1674a47e"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "57279f1ee810f9757f06663662719694"
server_key_iv: "cc8b8609b56574ff696e5859858c8d05"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.245223 31331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "0effc2046fa3492da9ef20dd1674a47e"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "57279f1ee810f9757f06663662719694"
server_key_iv: "cc8b8609b56574ff696e5859858c8d05"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.248759 31331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:19:10.251165 31346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.252426 31331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:10.252543 31331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "0effc2046fa3492da9ef20dd1674a47e"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "57279f1ee810f9757f06663662719694"
server_key_iv: "cc8b8609b56574ff696e5859858c8d05"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.252658 31331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:10.264461 31331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:10.267612 31331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:10.267818 31331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:10.268419 31331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:10.269340 31331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:10.269392 31331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.269433 31331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:10.269480 31331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.278281 31331 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:40229
I20260924 07:19:10.278338 31459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:40229 every 8 connection(s)
I20260924 07:19:10.279204 31331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:19:10.287523 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31331
I20260924 07:19:10.287669 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:19:10.287938 15870 external_mini_cluster.cc:1428] Setting key 7d0db534c23ad35f552c4c1c485bbcbe
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234350, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:10.294050 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.281265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:60301 (local address 127.15.127.190:37131)
0924 07:19:10.281409 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.281413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.282216 (+ 803us) server_negotiation.cc:407] Connection header received
0924 07:19:10.283502 (+ 1286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.283505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.283564 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.283657 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.284846 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.285454 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.286656 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.286944 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.287057 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.289944 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:10.289948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:10.289950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:10.289975 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:10.291761 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.292321 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.292327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.292329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.292419 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.292820 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.292822 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.292823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.292991 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:10.293140 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.293796 (+ 656us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.293926 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:19:10.295293 31462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.281625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:60301)
0924 07:19:10.282057 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.282100 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.283287 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.283849 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.283856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.284211 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:10.284689 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.284703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.285599 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.285602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.286526 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.286535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.287098 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.287102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.287294 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.288023 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:10.288049 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:10.289782 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:10.291899 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:10.291903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:10.291916 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:10.292178 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:10.292515 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:10.292521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:10.292526 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:10.292729 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:10.293155 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:10.293165 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:10.293489 (+ 324us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.293721 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.294118 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":101,"threads_started":1}
I20260924 07:19:10.296692 31460 heartbeater.cc:347] Connected to a master server at 127.15.127.190:37131
I20260924 07:19:10.296945 31460 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:10.297468 31460 heartbeater.cc:511] Master 127.15.127.190:37131 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:19:10.299008 31269 ts_manager.cc:194] Registered new tserver with Master: 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229)
I20260924 07:19:10.300217 31269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:60301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:19:10.317246 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:37131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:10.430137 31465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:10.430406 31465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:10.430480 31465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:10.434630 31465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:10.434687 31465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:10.434792 31465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:19:10.440277 31465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:10.441385 31465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:10.442447 31465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:10.451956 31473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:10.451964 31470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:10.452200 31465 server_base.cc:1034] running on GCE node
W20260924 07:19:10.451964 31471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:10.452633 31465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:10.453337 31465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:10.454519 31465 hybrid_clock.cc:648] HybridClock initialized: now 1790234350454507 us; error 25 us; skew 500 ppm
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234350, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:10.457535 31465 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:19:10.458386 31465 webserver.cc:460] Webserver started at http://127.15.127.130:38573/ using document root <none> and password file <none>
I20260924 07:19:10.458950 31465 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:10.459040 31465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:10.459270 31465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:10.461347 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "02112738b7614f269a42eaf9085d29d2"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "7e87452e0df988060561f357aabc4701"
server_key_iv: "b618fd1b2142e9b6688408d85568bce8"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.461925 31465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "02112738b7614f269a42eaf9085d29d2"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "7e87452e0df988060561f357aabc4701"
server_key_iv: "b618fd1b2142e9b6688408d85568bce8"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.465758 31465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:19:10.468152 31480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.469403 31465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:19:10.469533 31465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "02112738b7614f269a42eaf9085d29d2"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "7e87452e0df988060561f357aabc4701"
server_key_iv: "b618fd1b2142e9b6688408d85568bce8"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.469653 31465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:10.484627 31465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:10.487591 31465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:10.487775 31465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:10.488322 31465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:10.489143 31465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:10.489192 31465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.489230 31465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:10.489251 31465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.497779 31465 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:36879
I20260924 07:19:10.497802 31593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:36879 every 8 connection(s)
I20260924 07:19:10.498665 31465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:19:10.503662 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31465
I20260924 07:19:10.503798 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:19:10.504106 15870 external_mini_cluster.cc:1428] Setting key 54ad6f0427d3a22c2f4bd97d80966d2b
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234350, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:10.512972 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.500318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:54011 (local address 127.15.127.190:37131)
0924 07:19:10.500454 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.500457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.501223 (+ 766us) server_negotiation.cc:407] Connection header received
0924 07:19:10.502496 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.502500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.502576 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.502682 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.503987 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.504741 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.505963 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.506199 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.506309 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.509212 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:10.509218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:10.509224 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:10.509258 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:10.510898 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.511390 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.511395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.511401 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.511510 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.511832 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.511838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.511842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.512093 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:10.512197 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.512644 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.512773 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:19:10.513684 31596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.500633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:54011)
0924 07:19:10.501077 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.501118 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.502272 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.502850 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.502857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.503201 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:10.503838 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.503860 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.504868 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.504871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.505840 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.505849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.506350 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.506353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.506545 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.507238 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:10.507265 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:10.509048 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:10.511011 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:10.511016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:10.511027 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:10.511274 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:10.511602 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:10.511604 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:10.511605 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:10.511730 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:10.512191 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:10.512196 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:10.512400 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.512537 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.512778 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:19:10.514819 31594 heartbeater.cc:347] Connected to a master server at 127.15.127.190:37131
I20260924 07:19:10.515024 31594 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:10.515786 31594 heartbeater.cc:511] Master 127.15.127.190:37131 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:19:10.516947 31269 ts_manager.cc:194] Registered new tserver with Master: 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
I20260924 07:19:10.517630 31269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:54011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:19:10.532224 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:37131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:10.643814 31599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:10.644122 31599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:10.644167 31599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:10.648628 31599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:10.648691 31599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:10.648788 31599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:19:10.654275 31599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:10.655334 31599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:10.656499 31599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:19:10.666244 31599 server_base.cc:1034] running on GCE node
W20260924 07:19:10.666262 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:10.666301 31607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:10.666206 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:10.666880 31599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:10.667510 31599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:10.668682 31599 hybrid_clock.cc:648] HybridClock initialized: now 1790234350668666 us; error 27 us; skew 500 ppm
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234350, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:10.671321 31599 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:19:10.672209 31599 webserver.cc:460] Webserver started at http://127.15.127.131:42985/ using document root <none> and password file <none>
I20260924 07:19:10.672782 31599 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:10.672844 31599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:10.673170 31599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:19:10.675189 31599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "9ba6fb22b5824c828e0e95cc71b36ac8"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "ac6b23f76857815d841337659baa8878"
server_key_iv: "989fb18bad96e9daa6fe0afea9b8497a"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.675757 31599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "9ba6fb22b5824c828e0e95cc71b36ac8"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "ac6b23f76857815d841337659baa8878"
server_key_iv: "989fb18bad96e9daa6fe0afea9b8497a"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.679517 31599 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:19:10.681917 31614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.683115 31599 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:19:10.683257 31599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "9ba6fb22b5824c828e0e95cc71b36ac8"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "ac6b23f76857815d841337659baa8878"
server_key_iv: "989fb18bad96e9daa6fe0afea9b8497a"
server_key_version: "encryptionkey@0"
I20260924 07:19:10.683368 31599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:10.699009 31599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:10.702023 31599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:10.702224 31599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:10.702793 31599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:10.703588 31599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:10.703655 31599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.703724 31599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:10.703775 31599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:10.712931 31599 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:41031
I20260924 07:19:10.712954 31727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:41031 every 8 connection(s)
I20260924 07:19:10.713888 31599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:19:10.718555 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31599
I20260924 07:19:10.718645 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:19:10.718902 15870 external_mini_cluster.cc:1428] Setting key 864109dd427dab77ae391d4fb180a252
Sep 24 07:19:10 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234350, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:10.727484 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.715710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:42385 (local address 127.15.127.190:37131)
0924 07:19:10.715840 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.715850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.716734 (+ 884us) server_negotiation.cc:407] Connection header received
0924 07:19:10.718049 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.718053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.718125 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.718207 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.719588 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.720302 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.721538 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.721780 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.721892 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.724197 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:10.724201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:10.724204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:10.724230 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:10.725495 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.726004 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.726008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.726010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.726099 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.726429 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.726434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.726436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.726619 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:10.726726 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.727201 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.727319 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:19:10.728262 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.716032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:42385)
0924 07:19:10.716545 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.716611 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.717832 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.718388 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.718395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.718816 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:10.719442 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.719458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.720422 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.720425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.721413 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.721423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.721928 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.721933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.722110 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.722654 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:10.722669 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:10.724036 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:10.725631 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:10.725637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:10.725652 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:10.725898 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:10.726192 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:10.726197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:10.726199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:10.726333 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:10.726722 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:10.726728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:10.726971 (+ 243us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.727161 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.727470 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":102,"threads_started":1}
I20260924 07:19:10.729382 31728 heartbeater.cc:347] Connected to a master server at 127.15.127.190:37131
I20260924 07:19:10.729652 31728 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:10.730123 31728 heartbeater.cc:511] Master 127.15.127.190:37131 requested a full tablet report, sending...
I20260924 07:19:10.731153 31269 ts_manager.cc:194] Registered new tserver with Master: 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031)
I20260924 07:19:10.731716 31269 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:42385
I20260924 07:19:10.732749 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:19:10.741920 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.734886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.15.127.190:37131)
0924 07:19:10.735024 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.735030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.735221 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:19:10.735302 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.735304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.735341 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.735403 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:10.735981 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.736463 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.737423 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.737632 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.737709 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.739000 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:10.739003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:10.739005 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:10.739027 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:10.740435 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.740804 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.740806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.740808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.740877 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:10.741168 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:10.741170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:10.741172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:10.741372 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:10.741472 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.741638 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.741725 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:19:10.744521 31269 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:19:10.747568 31269 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:19:10.762343 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.757160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45968 (local address 127.15.127.130:36879)
0924 07:19:10.757546 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.757557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.757578 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:19:10.757647 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.757655 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.757840 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.758221 (+ 381us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:10.758383 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.759565 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.761105 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.761650 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.761786 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.761929 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.762093 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.762186 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":97,"threads_started":1}
I20260924 07:19:10.762374 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.757278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40229 (local address 127.0.0.1:36354)
0924 07:19:10.757571 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.757585 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.757670 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.758486 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.758489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.758507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:10.758600 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.758606 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.759881 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.759884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.760854 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.760865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.761936 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.761942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.762029 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.762054 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.762120 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.762198 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":84,"thread_start_us":144,"threads_started":1}
I20260924 07:19:10.762358 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.756838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.0.0.1:45968)
0924 07:19:10.757120 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.757133 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.757231 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.757980 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.757983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.757993 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:10.758274 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.758285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.759701 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.759704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.760854 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.760865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.761774 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.761777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.761879 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.761923 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.762056 (+ 133us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.762175 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":128,"thread_start_us":98,"threads_started":1}
I20260924 07:19:10.762868 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.757762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.15.127.129:40229)
0924 07:19:10.758026 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.758051 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.758076 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:19:10.758135 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.758139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.758340 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.758512 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:10.758695 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.759743 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.760980 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.761806 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.762458 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.762599 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.762646 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.762734 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260924 07:19:10.763865 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.756104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:41031 (local address 127.0.0.1:52288)
0924 07:19:10.756739 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.756790 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.756915 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.757957 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.757966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.757990 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:10.758225 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.758233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.760691 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.760694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.761711 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.761722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.763467 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.763471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.763571 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.763594 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.763674 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.763743 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":72,"spinlock_wait_cycles":1408,"thread_start_us":73,"threads_started":1}
I20260924 07:19:10.763916 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.756220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.15.127.131:41031)
0924 07:19:10.756843 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.756857 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.756884 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:19:10.757370 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.757377 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.757608 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.757762 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:10.758362 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.759578 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.761860 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.763363 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.763485 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.763603 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.763695 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.763776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":106,"threads_started":1}
I20260924 07:19:10.765707 31394 tablet_service.cc:1467] Processing CreateTablet for tablet 9036576351964af08bd2df9cfe808731 (DEFAULT_TABLE table=test-table [id=a17362339f174488af94e5a4de22a5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:10.765654 31528 tablet_service.cc:1467] Processing CreateTablet for tablet 9036576351964af08bd2df9cfe808731 (DEFAULT_TABLE table=test-table [id=a17362339f174488af94e5a4de22a5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:10.766780 31528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9036576351964af08bd2df9cfe808731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:10.766783 31394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9036576351964af08bd2df9cfe808731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:10.766745 31662 tablet_service.cc:1467] Processing CreateTablet for tablet 9036576351964af08bd2df9cfe808731 (DEFAULT_TABLE table=test-table [id=a17362339f174488af94e5a4de22a5c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:10.767607 31662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9036576351964af08bd2df9cfe808731. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:10.772917 31745 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Bootstrap starting.
I20260924 07:19:10.774583 31746 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap starting.
I20260924 07:19:10.775573 31745 tablet_bootstrap.cc:654] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:10.776314 31746 tablet_bootstrap.cc:654] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:10.776404 31745 log.cc:824] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:10.777186 31747 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Bootstrap starting.
I20260924 07:19:10.777588 31746 log.cc:824] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:10.778605 31745 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: No bootstrap required, opened a new log
I20260924 07:19:10.778787 31745 ts_tablet_manager.cc:1397] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:19:10.779127 31747 tablet_bootstrap.cc:654] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:10.779569 31746 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: No bootstrap required, opened a new log
I20260924 07:19:10.779752 31746 ts_tablet_manager.cc:1397] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:19:10.780164 31747 log.cc:824] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:10.781829 31747 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: No bootstrap required, opened a new log
I20260924 07:19:10.782019 31747 ts_tablet_manager.cc:1397] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:19:10.782704 31745 raft_consensus.cc:348] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.782996 31745 raft_consensus.cc:374] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:10.783087 31745 raft_consensus.cc:729] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02112738b7614f269a42eaf9085d29d2, State: Initialized, Role: FOLLOWER
I20260924 07:19:10.783605 31745 consensus_queue.cc:260] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.783520 31746 raft_consensus.cc:348] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.783749 31746 raft_consensus.cc:374] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:10.783793 31746 raft_consensus.cc:729] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0effc2046fa3492da9ef20dd1674a47e, State: Initialized, Role: FOLLOWER
I20260924 07:19:10.784251 31746 consensus_queue.cc:260] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.784603 31745 ts_tablet_manager.cc:1428] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:19:10.784919 31460 heartbeater.cc:503] Master 127.15.127.190:37131 was elected leader, sending a full tablet report...
I20260924 07:19:10.784955 31594 heartbeater.cc:503] Master 127.15.127.190:37131 was elected leader, sending a full tablet report...
I20260924 07:19:10.785226 31746 ts_tablet_manager.cc:1428] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260924 07:19:10.785466 31747 raft_consensus.cc:348] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.785749 31747 raft_consensus.cc:374] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:10.785815 31747 raft_consensus.cc:729] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba6fb22b5824c828e0e95cc71b36ac8, State: Initialized, Role: FOLLOWER
I20260924 07:19:10.786229 31747 consensus_queue.cc:260] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.786839 31728 heartbeater.cc:503] Master 127.15.127.190:37131 was elected leader, sending a full tablet report...
I20260924 07:19:10.787184 31747 ts_tablet_manager.cc:1428] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:19:10.797925 31753 raft_consensus.cc:482] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:10.798110 31753 raft_consensus.cc:504] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.799117 31753 leader_election.cc:288] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879), 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229)
I20260924 07:19:10.802738 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.799333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.15.127.131:32931)
0924 07:19:10.799454 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.799469 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.799558 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.799843 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.799846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.799863 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:10.799981 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.799987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.800717 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.800720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.801560 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.801570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.802303 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.802306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.802415 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.802440 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.802511 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.802589 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:19:10.802752 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.799473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:32931 (local address 127.15.127.130:36879)
0924 07:19:10.799618 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.799623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.799637 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:19:10.799685 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.799693 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.799731 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.799832 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:10.800087 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.800604 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.801693 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.802190 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.802309 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.802361 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.802562 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.802638 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:19:10.803330 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.799333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40229 (local address 127.15.127.131:32867)
0924 07:19:10.799837 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.799849 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:10.799964 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:10.800225 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:10.800228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:10.800243 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:10.800363 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.800368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.801114 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.801120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.801961 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:10.801978 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.802907 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.802914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:10.802992 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.803038 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:19:10.803145 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:19:10.803211 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":161,"thread_start_us":150,"threads_started":1}
I20260924 07:19:10.803355 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.799445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:32867 (local address 127.15.127.129:40229)
0924 07:19:10.799553 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.799568 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.799927 (+ 359us) server_negotiation.cc:407] Connection header received
0924 07:19:10.800084 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.800087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.800135 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.800218 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:10.800466 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.800951 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.802094 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.802636 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.802737 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.802793 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.803173 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.803249 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:19:10.803397 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9036576351964af08bd2df9cfe808731" candidate_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02112738b7614f269a42eaf9085d29d2" is_pre_election: true
I20260924 07:19:10.803681 31548 raft_consensus.cc:2454] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba6fb22b5824c828e0e95cc71b36ac8 in term 0.
I20260924 07:19:10.804171 31616 leader_election.cc:302] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8; no voters:
I20260924 07:19:10.804207 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9036576351964af08bd2df9cfe808731" candidate_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0effc2046fa3492da9ef20dd1674a47e" is_pre_election: true
I20260924 07:19:10.804419 31753 raft_consensus.cc:2781] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:19:10.804489 31414 raft_consensus.cc:2454] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ba6fb22b5824c828e0e95cc71b36ac8 in term 0.
I20260924 07:19:10.804524 31753 raft_consensus.cc:482] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:19:10.804607 31753 raft_consensus.cc:3037] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:10.805732 31753 raft_consensus.cc:504] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.806042 31753 leader_election.cc:288] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [CANDIDATE]: Term 1 election: Requested vote from peers 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879), 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229)
I20260924 07:19:10.806382 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9036576351964af08bd2df9cfe808731" candidate_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02112738b7614f269a42eaf9085d29d2"
I20260924 07:19:10.806404 31414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9036576351964af08bd2df9cfe808731" candidate_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0effc2046fa3492da9ef20dd1674a47e"
I20260924 07:19:10.806506 31548 raft_consensus.cc:3037] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:10.806506 31414 raft_consensus.cc:3037] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:10.807785 31414 raft_consensus.cc:2454] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba6fb22b5824c828e0e95cc71b36ac8 in term 1.
I20260924 07:19:10.807785 31548 raft_consensus.cc:2454] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ba6fb22b5824c828e0e95cc71b36ac8 in term 1.
I20260924 07:19:10.808079 31616 leader_election.cc:302] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8; no voters:
I20260924 07:19:10.808245 31753 raft_consensus.cc:2781] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:10.808447 31753 raft_consensus.cc:686] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 LEADER]: Becoming Leader. State: Replica: 9ba6fb22b5824c828e0e95cc71b36ac8, State: Running, Role: LEADER
I20260924 07:19:10.808763 31753 consensus_queue.cc:237] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:10.811887 31269 catalog_manager.cc:5495] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131). New cstate: current_term: 1 leader_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } health_report { overall_health: UNKNOWN } } }
I20260924 07:19:10.822819 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.819216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.15.127.131:41031)
0924 07:19:10.819328 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:10.819334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:10.819368 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:19:10.819555 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:10.819557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:10.819620 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:10.819729 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:19:10.819963 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.820454 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.821404 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:10.821620 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:10.821726 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:10.822018 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:19:10.822430 (+ 412us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:19:10.822560 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:10.822628 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:19:10.822680 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:19:10.828299 31414 raft_consensus.cc:1261] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Refusing update from remote peer 9ba6fb22b5824c828e0e95cc71b36ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:19:10.828299 31548 raft_consensus.cc:1261] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Refusing update from remote peer 9ba6fb22b5824c828e0e95cc71b36ac8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:19:10.829048 31755 consensus_queue.cc:1035] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:19:10.829289 31753 consensus_queue.cc:1035] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:19:10.836894 31759 mvcc.cc:205] Tried to move back new op lower bound from 7332799900986032128 to 7332799900915572736. Current Snapshot: MvccSnapshot[applied={T|T < 7332799900986032128}]
I20260924 07:19:10.845793 31268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:19:10.845973 31268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:19:10.848367 31268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Sending DeleteTablet for 3 replicas of tablet 9036576351964af08bd2df9cfe808731
I20260924 07:19:10.849234 31394 tablet_service.cc:1514] Processing DeleteTablet for tablet 9036576351964af08bd2df9cfe808731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:36354
I20260924 07:19:10.849233 31662 tablet_service.cc:1514] Processing DeleteTablet for tablet 9036576351964af08bd2df9cfe808731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:52288
I20260924 07:19:10.849452 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31331
I20260924 07:19:10.849721 31528 tablet_service.cc:1514] Processing DeleteTablet for tablet 9036576351964af08bd2df9cfe808731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:45968
I20260924 07:19:10.851326 31768 tablet_replica.cc:330] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: stopping tablet replica
I20260924 07:19:10.851646 31768 raft_consensus.cc:2229] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:19:10.851951 31768 raft_consensus.cc:2258] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:10.853224 31767 tablet_replica.cc:330] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: stopping tablet replica
I20260924 07:19:10.853382 31768 ts_tablet_manager.cc:1905] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:10.853586 31767 raft_consensus.cc:2229] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:19:10.853978 31767 raft_consensus.cc:2258] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:10.855525 31767 ts_tablet_manager.cc:1905] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:19:10.856778 31253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:40229 (error 108)
I20260924 07:19:10.857015 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31465
I20260924 07:19:10.857126 31768 ts_tablet_manager.cc:1918] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:10.857218 31768 log.cc:1196] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/9036576351964af08bd2df9cfe808731
I20260924 07:19:10.860013 31767 ts_tablet_manager.cc:1918] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:10.860133 31767 log.cc:1196] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/9036576351964af08bd2df9cfe808731
I20260924 07:19:10.860431 31767 ts_tablet_manager.cc:1939] T 9036576351964af08bd2df9cfe808731 P 9ba6fb22b5824c828e0e95cc71b36ac8: Deleting consensus metadata
I20260924 07:19:10.861860 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.861539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40229 (local address 127.0.0.1:36358)
0924 07:19:10.861678 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.861764 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:19:10.861887 31254 catalog_manager.cc:4841] TS 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): tablet 9036576351964af08bd2df9cfe808731 (table test-table [id=a17362339f174488af94e5a4de22a5c1]) successfully deleted
W20260924 07:19:10.862105 31253 catalog_manager.cc:4584] TS 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9036576351964af08bd2df9cfe808731: Network error: Client connection negotiation failed: client connection to 127.15.127.129:40229: connect: Connection refused (error 111)
I20260924 07:19:10.864513 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31599
I20260924 07:19:10.865639 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:10.864648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.0.0.1:45972)
0924 07:19:10.865513 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:10.865582 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:36879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":799}
I20260924 07:19:10.869570 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31237
I20260924 07:19:30.876271 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:37131
--webserver_interface=127.15.127.190
--webserver_port=36035
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:30.986056 31770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:30.986343 31770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:30.986385 31770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:30.990507 31770 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:19:30.990569 31770 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:19:30.990585 31770 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:19:30.990600 31770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:30.990644 31770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:30.990676 31770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:30.990702 31770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:30.990738 31770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:30.996119 31770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=36035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:30.997254 31770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:30.998277 31770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:19:31.011797 31770 server_base.cc:1034] running on GCE node
W20260924 07:19:31.011754 31776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.011771 31779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.011754 31777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:31.012717 31770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:31.013965 31770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:31.015151 31770 hybrid_clock.cc:648] HybridClock initialized: now 1790234371015117 us; error 29 us; skew 500 ppm
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:31.017961 31770 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:19:31.018886 31770 webserver.cc:460] Webserver started at http://127.15.127.190:36035/ using document root <none> and password file <none>
I20260924 07:19:31.019471 31770 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:31.019598 31770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:31.023568 31770 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:31.025820 31786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.027040 31770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:19:31.027189 31770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "72810508f1ef4581a529f869a1098fbb"
format_stamp: "Formatted at 2026-09-24 07:19:09 on dist-test-slave-zx46"
server_key: "694394066c374ced8b770f9080074384"
server_key_iv: "577747b9e01c896543a3419768203247"
server_key_version: "encryptionkey@0"
I20260924 07:19:31.027787 31770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:31.044147 31770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:31.047149 31770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:31.047355 31770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:31.054553 31770 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:37131
I20260924 07:19:31.054867 31838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:37131 every 8 connection(s)
I20260924 07:19:31.055653 31770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:31.060140 31839 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:19:31.061864 31839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap starting.
I20260924 07:19:31.062884 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31770
I20260924 07:19:31.063411 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:40229
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:37131
--webserver_port=34811
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:19:31.080426 31839 log.cc:824] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:31.089814 31839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:19:31.090224 31839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap complete.
I20260924 07:19:31.094352 31839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:31.094735 31839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72810508f1ef4581a529f869a1098fbb, State: Initialized, Role: FOLLOWER
I20260924 07:19:31.095235 31839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:31.095410 31839 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:31.095517 31839 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:31.095651 31839 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:19:31.097707 31839 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:31.098057 31839 leader_election.cc:302] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72810508f1ef4581a529f869a1098fbb; no voters:
I20260924 07:19:31.098455 31839 leader_election.cc:288] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:19:31.098505 31843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:19:31.098714 31843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 LEADER]: Becoming Leader. State: Replica: 72810508f1ef4581a529f869a1098fbb, State: Running, Role: LEADER
I20260924 07:19:31.099097 31843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:31.099713 31839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:31.100615 31844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72810508f1ef4581a529f869a1098fbb. Latest consensus state: current_term: 2 leader_uuid: "72810508f1ef4581a529f869a1098fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } } }
I20260924 07:19:31.100742 31844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:31.101116 31851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:31.101125 31845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72810508f1ef4581a529f869a1098fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } } }
I20260924 07:19:31.101253 31845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:31.106174 31851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:31.106877 31851 catalog_manager.cc:1259] Loaded cluster ID: 791312e0b861401cb76309251784bbc1
I20260924 07:19:31.106945 31851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:19:31.109483 31851 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:31.110172 31851 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Loaded TSK: 0
I20260924 07:19:31.129110 31851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Generated new TSK 1
I20260924 07:19:31.129413 31851 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:19:31.176944 31841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:31.177263 31841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:31.177338 31841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:31.181329 31841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:31.181416 31841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:31.181510 31841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:19:31.186636 31841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:40229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=34811
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:31.187700 31841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:31.188726 31841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:31.199258 31866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.199368 31869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.199254 31867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:31.199623 31841 server_base.cc:1034] running on GCE node
I20260924 07:19:31.200069 31841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:31.200704 31841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:31.201887 31841 hybrid_clock.cc:648] HybridClock initialized: now 1790234371201847 us; error 57 us; skew 500 ppm
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:31.204958 31841 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:19:31.206056 31841 webserver.cc:460] Webserver started at http://127.15.127.129:34811/ using document root <none> and password file <none>
I20260924 07:19:31.206738 31841 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:31.206839 31841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:31.211813 31841 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260924 07:19:31.214462 31876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.215963 31841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 07:19:31.216095 31841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "0effc2046fa3492da9ef20dd1674a47e"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "57279f1ee810f9757f06663662719694"
server_key_iv: "cc8b8609b56574ff696e5859858c8d05"
server_key_version: "encryptionkey@0"
I20260924 07:19:31.216780 31841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:31.243185 31841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:31.247094 31841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:31.247298 31841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:31.247988 31841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:31.249372 31883 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:19:31.253602 31841 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:19:31.253687 31841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 07:19:31.253759 31841 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:19:31.256136 31841 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:19:31.256206 31841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 07:19:31.256487 31883 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap starting.
I20260924 07:19:31.267658 31841 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:40229
I20260924 07:19:31.268416 31990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:40229 every 8 connection(s)
I20260924 07:19:31.269071 31841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:19:31.270402 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31841
I20260924 07:19:31.271013 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:36879
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:37131
--webserver_port=38573
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:19:31.277467 31883 log.cc:824] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Log is configured to *not* fsync() on all Append() calls
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:31.285236 31883 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:19:31.285692 31883 tablet_bootstrap.cc:492] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap complete.
I20260924 07:19:31.286981 31883 ts_tablet_manager.cc:1397] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.007s
I20260924 07:19:31.289156 31993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.271508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:58081)
0924 07:19:31.272055 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.272109 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.273155 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.274647 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.274659 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.275024 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:31.275490 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.275505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.277322 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.277329 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.278932 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.278948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.279928 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.279933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.280183 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.280927 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:31.280954 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:31.283093 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:31.285832 (+ 2739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.285838 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.285851 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.286181 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.286583 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.286586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.286588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.286749 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.287430 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:31.287436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:31.287718 (+ 282us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.287914 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.288220 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260924 07:19:31.289186 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.272768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:58081 (local address 127.15.127.190:37131)
0924 07:19:31.273180 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.273199 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.273244 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:19:31.273630 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.273649 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.274155 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.274540 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:31.275683 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.277130 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.279081 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.279764 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.279939 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.283329 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:31.283336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:31.283351 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:31.283385 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:31.285681 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.286304 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.286313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.286322 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.286473 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.286848 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.286854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.286859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.287312 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:31.287447 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.288065 (+ 618us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.288352 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":113,"threads_started":1}
I20260924 07:19:31.289778 31883 raft_consensus.cc:348] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.290243 31883 raft_consensus.cc:729] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0effc2046fa3492da9ef20dd1674a47e, State: Initialized, Role: FOLLOWER
I20260924 07:19:31.290767 31883 consensus_queue.cc:260] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.291764 31883 ts_tablet_manager.cc:1428] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:19:31.292140 31991 heartbeater.cc:347] Connected to a master server at 127.15.127.190:37131
I20260924 07:19:31.292359 31991 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:31.292825 31991 heartbeater.cc:511] Master 127.15.127.190:37131 requested a full tablet report, sending...
I20260924 07:19:31.295320 31803 ts_manager.cc:194] Registered new tserver with Master: 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229)
I20260924 07:19:31.299263 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:58081
I20260924 07:19:31.301892 31991 heartbeater.cc:503] Master 127.15.127.190:37131 was elected leader, sending a full tablet report...
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.129@KRBTEST.COM
I20260924 07:19:31.309600 31999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.298748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.15.127.129:40229)
0924 07:19:31.299394 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.299407 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.299768 (+ 361us) server_negotiation.cc:407] Connection header received
0924 07:19:31.299986 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.299991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.300262 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.300431 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:31.301149 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.302178 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.303044 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.303479 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.303586 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.306019 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:31.306028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:31.306037 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:31.306058 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:31.307664 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.308158 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.308164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.308165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.308257 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.308540 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.308544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.308546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.308768 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:31.308904 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.309301 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.309458 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":91,"threads_started":1}
I20260924 07:19:31.309600 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.298684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40229 (local address 127.0.0.1:34234)
0924 07:19:31.299626 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.299659 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.299835 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.300408 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.300414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.300837 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:31.300958 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.300965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.302306 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.302309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.302918 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.302927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.303600 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.303604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.303712 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.304281 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:31.304295 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:31.305867 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:31.307790 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.307799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.307804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.308039 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.308346 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.308350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.308352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.308436 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.308878 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:31.308884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:31.309049 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.309195 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.309405 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":118,"threads_started":1}
I20260924 07:19:31.310556 31925 tablet_service.cc:1514] Processing DeleteTablet for tablet 9036576351964af08bd2df9cfe808731 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:34234
I20260924 07:19:31.311599 32000 tablet_replica.cc:330] stopping tablet replica
I20260924 07:19:31.311901 32000 raft_consensus.cc:2229] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:19:31.312139 32000 raft_consensus.cc:2258] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:31.313099 32000 ts_tablet_manager.cc:1905] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:31.316413 32000 ts_tablet_manager.cc:1918] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:31.316538 32000 log.cc:1196] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/9036576351964af08bd2df9cfe808731
I20260924 07:19:31.316809 32000 ts_tablet_manager.cc:1939] T 9036576351964af08bd2df9cfe808731 P 0effc2046fa3492da9ef20dd1674a47e: Deleting consensus metadata
I20260924 07:19:31.318464 31787 catalog_manager.cc:4841] TS 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229): tablet 9036576351964af08bd2df9cfe808731 (table test-table [id=a17362339f174488af94e5a4de22a5c1]) successfully deleted
W20260924 07:19:31.391724 31994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:31.391965 31994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:31.392007 31994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:31.395965 31994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:31.396021 31994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:31.396083 31994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:19:31.401379 31994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:36879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=38573
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:31.402369 31994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:31.403287 31994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:31.412541 32006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.412616 32008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.412567 32005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:31.412838 31994 server_base.cc:1034] running on GCE node
I20260924 07:19:31.413383 31994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:31.414026 31994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:31.415215 31994 hybrid_clock.cc:648] HybridClock initialized: now 1790234371415185 us; error 49 us; skew 500 ppm
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:31.417877 31994 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:19:31.418769 31994 webserver.cc:460] Webserver started at http://127.15.127.130:38573/ using document root <none> and password file <none>
I20260924 07:19:31.419338 31994 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:31.419445 31994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:31.423525 31994 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:19:31.426025 32015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.427237 31994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:19:31.427371 31994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "02112738b7614f269a42eaf9085d29d2"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "7e87452e0df988060561f357aabc4701"
server_key_iv: "b618fd1b2142e9b6688408d85568bce8"
server_key_version: "encryptionkey@0"
I20260924 07:19:31.427911 31994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:31.452206 31994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:31.455119 31994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:31.455363 31994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:31.455953 31994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:31.456990 32022 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 07:19:31.460729 32022 ts_tablet_manager.cc:1852] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 07:19:31.461195 32022 ts_tablet_manager.cc:1905] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:31.465405 32022 ts_tablet_manager.cc:1918] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:31.465579 32022 ts_tablet_manager.cc:1939] T 9036576351964af08bd2df9cfe808731 P 02112738b7614f269a42eaf9085d29d2: Deleting consensus metadata
I20260924 07:19:31.465910 31994 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 07:19:31.465987 31994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260924 07:19:31.466085 31994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:31.466137 31994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.474895 31994 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:36879
I20260924 07:19:31.474933 32129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:36879 every 8 connection(s)
I20260924 07:19:31.475755 31994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:19:31.480057 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 31994
I20260924 07:19:31.480465 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:41031
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:37131
--webserver_port=42985
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:31.490339 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.477521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:55715 (local address 127.15.127.190:37131)
0924 07:19:31.477693 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.477698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.478515 (+ 817us) server_negotiation.cc:407] Connection header received
0924 07:19:31.479404 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.479407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.479456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.479535 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:31.480805 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.481668 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.483428 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.483688 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.483779 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.486698 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:31.486701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:31.486703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:31.486728 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:31.488202 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.488704 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.488708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.488710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.488787 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.489217 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.489222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.489223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.489416 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:31.489527 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.490030 (+ 503us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.490163 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 07:19:31.491139 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.477929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:55715)
0924 07:19:31.478361 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.478402 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.479218 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.479703 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.479709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.480038 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:31.480585 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.480616 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.481787 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.481790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.483230 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.483244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.483851 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.483858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.484101 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.484735 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:31.484761 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:31.486519 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:31.488317 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.488323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.488338 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.488602 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.488884 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.488888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.488890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.489098 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.489531 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:31.489540 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:31.489789 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.489981 (+ 192us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.490300 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":101,"threads_started":1}
I20260924 07:19:31.492304 32130 heartbeater.cc:347] Connected to a master server at 127.15.127.190:37131
I20260924 07:19:31.492547 32130 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:31.493060 32130 heartbeater.cc:511] Master 127.15.127.190:37131 requested a full tablet report, sending...
I20260924 07:19:31.494134 31803 ts_manager.cc:194] Registered new tserver with Master: 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
I20260924 07:19:31.494872 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:55715
W20260924 07:19:31.592263 32133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:31.592510 32133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:31.592569 32133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:31.596673 32133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:31.596731 32133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:31.596815 32133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:19:31.602424 32133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:41031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=42985
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:31.603404 32133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:31.604372 32133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:31.613610 32138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:31.613773 32133 server_base.cc:1034] running on GCE node
W20260924 07:19:31.613692 32139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:31.613688 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:31.614218 32133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:31.615043 32133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:31.616220 32133 hybrid_clock.cc:648] HybridClock initialized: now 1790234371616207 us; error 28 us; skew 500 ppm
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:31.618865 32133 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:19:31.619732 32133 webserver.cc:460] Webserver started at http://127.15.127.131:42985/ using document root <none> and password file <none>
I20260924 07:19:31.620234 32133 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:31.620299 32133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:31.624063 32133 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:31.626053 32148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.627311 32133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:31.627398 32133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "9ba6fb22b5824c828e0e95cc71b36ac8"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "ac6b23f76857815d841337659baa8878"
server_key_iv: "989fb18bad96e9daa6fe0afea9b8497a"
server_key_version: "encryptionkey@0"
I20260924 07:19:31.627920 32133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:31.638937 32133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:31.641651 32133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:31.641796 32133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:31.642300 32133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:31.643050 32133 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:31.643096 32133 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.643133 32133 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:31.643149 32133 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:31.651995 32133 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:41031
I20260924 07:19:31.652077 32261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:41031 every 8 connection(s)
I20260924 07:19:31.652980 32133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:19:31.657850 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 32133
Sep 24 07:19:31 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234371, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:19:31.666842 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.658538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33046 (local address 127.15.127.190:37131)
0924 07:19:31.658776 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.658781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.658915 (+ 134us) server_negotiation.cc:407] Connection header received
0924 07:19:31.658998 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.659001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.659047 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.659131 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:31.659796 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.660637 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.661532 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.661896 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.662011 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.663392 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:31.663396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:31.663398 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:31.663420 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:31.665147 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.665599 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.665603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.665605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.665690 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.665965 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.665969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.665971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.666149 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:31.666358 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.666506 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.666627 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260924 07:19:31.667899 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.654923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:45713 (local address 127.15.127.190:37131)
0924 07:19:31.655066 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.655069 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.655994 (+ 925us) server_negotiation.cc:407] Connection header received
0924 07:19:31.657392 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.657395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.657463 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.657568 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:31.658998 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.659595 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.661181 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.661422 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.661536 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.664148 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:31.664151 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:31.664153 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:31.664184 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:31.665658 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.666220 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.666225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.666227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.666307 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.666669 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.666672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.666674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.666908 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:31.667019 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.667668 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.667767 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":52}
I20260924 07:19:31.668896 32264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.655294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:45713)
0924 07:19:31.655754 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.655794 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.657175 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.657746 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.657752 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.658198 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:31.658809 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.658835 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.659720 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.659722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.661070 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.661079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.661575 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.661582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.661841 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.662511 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:31.662536 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:31.663982 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:31.665781 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.665787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.665802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.666111 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.666408 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:31.666411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:31.666413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:31.666559 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:31.667072 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:31.667078 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:31.667398 (+ 320us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.667611 (+ 213us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.667971 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":108,"threads_started":1}
I20260924 07:19:31.669986 32262 heartbeater.cc:347] Connected to a master server at 127.15.127.190:37131
I20260924 07:19:31.670255 32262 heartbeater.cc:464] Registering TS with master...
I20260924 07:19:31.670722 32262 heartbeater.cc:511] Master 127.15.127.190:37131 requested a full tablet report, sending...
I20260924 07:19:31.671681 31803 ts_manager.cc:194] Registered new tserver with Master: 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031)
I20260924 07:19:31.672227 31803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:45713
I20260924 07:19:31.680434 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:19:31.684204 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.680988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.15.127.190:37131)
0924 07:19:31.681144 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.681148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.681341 (+ 193us) server_negotiation.cc:407] Connection header received
0924 07:19:31.681463 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.681465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.681569 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.681649 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:19:31.681848 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.682331 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.683148 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.683384 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.683475 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.683726 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:19:31.683852 (+ 126us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 07:19:31.684055 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33062: token expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 07:19:31.684273 31995 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33062: token expired
W20260924 07:19:31.684579 31733 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.127.190:37131)
I20260924 07:19:31.691304 31995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.684945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.15.127.190:37131)
0924 07:19:31.685099 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.685101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.685113 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:19:31.685189 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.685191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.685222 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.685281 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:31.685877 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.686333 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.687055 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.687247 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.687320 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.688519 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:31.688522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:31.688524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:31.688545 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:31.689950 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.690339 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.690341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.690342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.690409 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:31.690632 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:31.690634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:31.690635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:31.690778 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:31.690881 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.691054 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.691132 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 07:19:31.693557 31803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:19:31.696657 31803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:19:31.704567 31925 tablet_service.cc:1467] Processing CreateTablet for tablet ee813c97fd1c4c81a235505118c4b593 (DEFAULT_TABLE table=test-table [id=396c086a4fd54e6dac7b68120c1dd2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:31.705273 31925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee813c97fd1c4c81a235505118c4b593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:31.708173 31883 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap starting.
I20260924 07:19:31.709096 31883 tablet_bootstrap.cc:654] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:31.709316 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.703859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.0.0.1:49022)
0924 07:19:31.704141 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.704155 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.704236 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.705170 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.705173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.705191 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:31.705301 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.705308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.706444 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.706447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.707561 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.707571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.708791 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.708795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.708885 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.708956 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.709073 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.709167 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":23,"thread_start_us":120,"threads_started":1}
I20260924 07:19:31.709668 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.704228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45042 (local address 127.15.127.131:41031)
0924 07:19:31.704717 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.704731 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.704754 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:19:31.704967 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.704973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.705190 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.705621 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:31.705778 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.706833 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.708259 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.708819 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.708956 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.709106 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.709405 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.709525 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":159,"threads_started":1}
I20260924 07:19:31.709766 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.703616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:41031 (local address 127.0.0.1:45042)
0924 07:19:31.703741 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.703756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.703831 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.705352 (+ 1521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.705355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.705372 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:31.705458 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.705463 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.706961 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.706964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.708090 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.708101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.708948 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.708952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.709086 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.709107 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.709187 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.709281 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":3}
I20260924 07:19:31.710017 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.704242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.15.127.130:36879)
0924 07:19:31.704586 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.704616 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.704641 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:19:31.704722 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.704727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.704981 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.705470 (+ 489us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:31.705609 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.706320 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.707688 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.708651 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.709575 (+ 924us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.709702 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.709757 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.709840 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":159,"threads_started":1}
I20260924 07:19:31.710590 31883 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: No bootstrap required, opened a new log
I20260924 07:19:31.710667 31883 ts_tablet_manager.cc:1397] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:19:31.711088 31883 raft_consensus.cc:348] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.711210 31883 raft_consensus.cc:374] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:31.711246 31883 raft_consensus.cc:729] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0effc2046fa3492da9ef20dd1674a47e, State: Initialized, Role: FOLLOWER
I20260924 07:19:31.711369 31883 consensus_queue.cc:260] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.711611 31883 ts_tablet_manager.cc:1428] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:19:31.712533 32064 tablet_service.cc:1467] Processing CreateTablet for tablet ee813c97fd1c4c81a235505118c4b593 (DEFAULT_TABLE table=test-table [id=396c086a4fd54e6dac7b68120c1dd2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:31.713186 32196 tablet_service.cc:1467] Processing CreateTablet for tablet ee813c97fd1c4c81a235505118c4b593 (DEFAULT_TABLE table=test-table [id=396c086a4fd54e6dac7b68120c1dd2f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:19:31.713618 32064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee813c97fd1c4c81a235505118c4b593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:31.714193 32196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee813c97fd1c4c81a235505118c4b593. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:19:31.718122 32022 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Bootstrap starting.
I20260924 07:19:31.720193 32022 tablet_bootstrap.cc:654] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:31.720979 32022 log.cc:824] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:31.721565 32275 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Bootstrap starting.
I20260924 07:19:31.723246 32022 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: No bootstrap required, opened a new log
I20260924 07:19:31.723443 32022 ts_tablet_manager.cc:1397] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:19:31.723834 32275 tablet_bootstrap.cc:654] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Neither blocks nor log segments found. Creating new log.
I20260924 07:19:31.724506 32275 log.cc:824] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:31.726456 32275 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: No bootstrap required, opened a new log
I20260924 07:19:31.726658 32275 ts_tablet_manager.cc:1397] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:19:31.726998 32022 raft_consensus.cc:348] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.727200 32022 raft_consensus.cc:374] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:31.727255 32022 raft_consensus.cc:729] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02112738b7614f269a42eaf9085d29d2, State: Initialized, Role: FOLLOWER
I20260924 07:19:31.727721 32022 consensus_queue.cc:260] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.728364 32130 heartbeater.cc:503] Master 127.15.127.190:37131 was elected leader, sending a full tablet report...
I20260924 07:19:31.728675 32022 ts_tablet_manager.cc:1428] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:19:31.730053 32275 raft_consensus.cc:348] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.730338 32275 raft_consensus.cc:374] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:19:31.730399 32275 raft_consensus.cc:729] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ba6fb22b5824c828e0e95cc71b36ac8, State: Initialized, Role: FOLLOWER
I20260924 07:19:31.730818 32275 consensus_queue.cc:260] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.731560 32262 heartbeater.cc:503] Master 127.15.127.190:37131 was elected leader, sending a full tablet report...
I20260924 07:19:31.731787 32275 ts_tablet_manager.cc:1428] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 07:19:31.773921 31992 tablet.cc:2369] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:19:31.803849 31997 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:31.804083 31997 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.805346 31997 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
I20260924 07:19:31.809554 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.805558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:41031 (local address 127.15.127.129:54209)
0924 07:19:31.805800 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.805817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.805938 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.806402 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.806407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.806424 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:31.806543 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.806548 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.807313 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.807318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.808340 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.808349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.809044 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.809048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.809225 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.809285 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.809357 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.809425 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":49,"thread_start_us":91,"threads_started":1}
I20260924 07:19:31.809607 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.805666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:54209 (local address 127.15.127.131:41031)
0924 07:19:31.805800 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.805804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.805901 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:19:31.806135 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.806139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.806193 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.806244 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:31.806667 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.807195 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.808475 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.808881 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.809003 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.809097 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.809404 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.809486 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:19:31.810477 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.805921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.15.127.129:37439)
0924 07:19:31.806262 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.806275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:31.806376 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:31.807343 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:31.807345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:31.807365 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:19:31.807473 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.807479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.808229 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.808233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.809236 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:31.809246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.810152 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.810154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:31.810228 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.810244 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:19:31.810303 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:19:31.810351 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":64,"thread_start_us":82,"threads_started":1}
I20260924 07:19:31.810535 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.806134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:37439 (local address 127.15.127.130:36879)
0924 07:19:31.806261 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.806266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.806336 (+ 70us) server_negotiation.cc:407] Connection header received
0924 07:19:31.806482 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.806484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.806543 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.806612 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:19:31.807612 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.808096 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.809375 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.810042 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.810165 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.810210 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.810321 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.810400 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:19:31.810914 32216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee813c97fd1c4c81a235505118c4b593" candidate_uuid: "0effc2046fa3492da9ef20dd1674a47e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" is_pre_election: true
I20260924 07:19:31.811245 32216 raft_consensus.cc:2454] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0effc2046fa3492da9ef20dd1674a47e in term 0.
I20260924 07:19:31.811228 32084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee813c97fd1c4c81a235505118c4b593" candidate_uuid: "0effc2046fa3492da9ef20dd1674a47e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02112738b7614f269a42eaf9085d29d2" is_pre_election: true
I20260924 07:19:31.811524 32084 raft_consensus.cc:2454] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0effc2046fa3492da9ef20dd1674a47e in term 0.
I20260924 07:19:31.811774 31878 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0effc2046fa3492da9ef20dd1674a47e, 9ba6fb22b5824c828e0e95cc71b36ac8; no voters:
I20260924 07:19:31.812067 31997 raft_consensus.cc:2781] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:19:31.812142 31997 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:19:31.812225 31997 raft_consensus.cc:3037] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:31.813342 31997 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.813699 31997 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 1 election: Requested vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
I20260924 07:19:31.814112 32084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee813c97fd1c4c81a235505118c4b593" candidate_uuid: "0effc2046fa3492da9ef20dd1674a47e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02112738b7614f269a42eaf9085d29d2"
I20260924 07:19:31.814110 32216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee813c97fd1c4c81a235505118c4b593" candidate_uuid: "0effc2046fa3492da9ef20dd1674a47e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8"
I20260924 07:19:31.814246 32084 raft_consensus.cc:3037] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:31.814257 32216 raft_consensus.cc:3037] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:19:31.815269 32084 raft_consensus.cc:2454] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0effc2046fa3492da9ef20dd1674a47e in term 1.
I20260924 07:19:31.815289 32216 raft_consensus.cc:2454] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0effc2046fa3492da9ef20dd1674a47e in term 1.
I20260924 07:19:31.815569 31878 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02112738b7614f269a42eaf9085d29d2, 0effc2046fa3492da9ef20dd1674a47e; no voters:
I20260924 07:19:31.815729 31997 raft_consensus.cc:2781] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:19:31.815930 31997 raft_consensus.cc:686] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 LEADER]: Becoming Leader. State: Replica: 0effc2046fa3492da9ef20dd1674a47e, State: Running, Role: LEADER
I20260924 07:19:31.816210 31997 consensus_queue.cc:237] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:31.819106 31802 catalog_manager.cc:5495] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129). New cstate: current_term: 1 leader_uuid: "0effc2046fa3492da9ef20dd1674a47e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } health_report { overall_health: HEALTHY } } }
I20260924 07:19:31.837881 32285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.834562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34238 (local address 127.15.127.129:40229)
0924 07:19:31.834771 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:31.834774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:31.834786 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:19:31.834852 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:31.834854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:31.834903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:31.834998 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:19:31.835205 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.835711 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.836472 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:31.836696 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:31.836770 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:31.836988 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:19:31.837460 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:19:31.837563 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:31.837633 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:19:31.837679 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260924 07:19:31.841629 32216 raft_consensus.cc:1261] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Refusing update from remote peer 0effc2046fa3492da9ef20dd1674a47e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:19:31.841629 32084 raft_consensus.cc:1261] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Refusing update from remote peer 0effc2046fa3492da9ef20dd1674a47e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:19:31.842346 31997 consensus_queue.cc:1035] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:19:31.842581 32282 consensus_queue.cc:1035] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:19:31.850800 32287 mvcc.cc:205] Tried to move back new op lower bound from 7332799987057291264 to 7332799986961924096. Current Snapshot: MvccSnapshot[applied={T|T < 7332799987057291264}]
I20260924 07:19:31.856629 31801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:19:31.856796 31801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:19:31.859375 31801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Sending DeleteTablet for 3 replicas of tablet ee813c97fd1c4c81a235505118c4b593
I20260924 07:19:31.859947 31925 tablet_service.cc:1514] Processing DeleteTablet for tablet ee813c97fd1c4c81a235505118c4b593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:31 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:34234
I20260924 07:19:31.860224 32064 tablet_service.cc:1514] Processing DeleteTablet for tablet ee813c97fd1c4c81a235505118c4b593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:31 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:49022
I20260924 07:19:31.860272 32196 tablet_service.cc:1514] Processing DeleteTablet for tablet ee813c97fd1c4c81a235505118c4b593 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:19:31 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:45042
I20260924 07:19:31.860389 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31841
I20260924 07:19:31.862285 32297 tablet_replica.cc:330] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: stopping tablet replica
I20260924 07:19:31.862571 32297 raft_consensus.cc:2229] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:19:31.862843 32297 raft_consensus.cc:2258] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:31.863198 32296 tablet_replica.cc:330] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: stopping tablet replica
I20260924 07:19:31.863488 32296 raft_consensus.cc:2229] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:19:31.863758 32296 raft_consensus.cc:2258] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:19:31.864284 32297 ts_tablet_manager.cc:1905] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:31.865134 32296 ts_tablet_manager.cc:1905] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:31.867115 32297 ts_tablet_manager.cc:1918] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 07:19:31.867242 31787 connection.cc:535] client connection to 127.15.127.129:40229 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 07:19:31.867396 31787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:19:31.867616 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31994
I20260924 07:19:31.867709 32296 ts_tablet_manager.cc:1918] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:31.868221 32297 log.cc:1196] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/ee813c97fd1c4c81a235505118c4b593
I20260924 07:19:31.868539 32297 ts_tablet_manager.cc:1939] T ee813c97fd1c4c81a235505118c4b593 P 9ba6fb22b5824c828e0e95cc71b36ac8: Deleting consensus metadata
I20260924 07:19:31.869241 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.868853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40229 (local address 127.0.0.1:34244)
0924 07:19:31.869055 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.869137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:40229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 07:19:31.869879 31788 catalog_manager.cc:4841] TS 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): tablet ee813c97fd1c4c81a235505118c4b593 (table test-table [id=396c086a4fd54e6dac7b68120c1dd2f6]) successfully deleted
W20260924 07:19:31.869587 31787 catalog_manager.cc:4584] TS 0effc2046fa3492da9ef20dd1674a47e (127.15.127.129:40229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee813c97fd1c4c81a235505118c4b593: Network error: Client connection negotiation failed: client connection to 127.15.127.129:40229: connect: Connection refused (error 111)
I20260924 07:19:31.874225 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 32133
I20260924 07:19:31.876256 31998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:31.874347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.0.0.1:49024)
0924 07:19:31.876091 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:31.876158 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:36879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1639}
I20260924 07:19:31.879612 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 31770
I20260924 07:19:51.886137 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:37131
--webserver_interface=127.15.127.190
--webserver_port=36035
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:19:51.996496 32299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:51.996783 32299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:51.996858 32299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:52.000830 32299 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:19:52.000912 32299 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:19:52.000938 32299 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:19:52.000962 32299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:19:52.001013 32299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:52.001053 32299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:52.001111 32299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:19:52.001148 32299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:19:52.006227 32299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=36035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:52.007290 32299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:52.008294 32299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:52.017197 32304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.017236 32305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.017508 32307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:52.017571 32299 server_base.cc:1034] running on GCE node
I20260924 07:19:52.018203 32299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:52.019511 32299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:52.020696 32299 hybrid_clock.cc:648] HybridClock initialized: now 1790234392020660 us; error 55 us; skew 500 ppm
Sep 24 07:19:52 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:52.024130 32299 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:19:52.025170 32299 webserver.cc:460] Webserver started at http://127.15.127.190:36035/ using document root <none> and password file <none>
I20260924 07:19:52.025810 32299 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:52.025876 32299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:52.030365 32299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:19:52.032397 32314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.033669 32299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:52.033833 32299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "72810508f1ef4581a529f869a1098fbb"
format_stamp: "Formatted at 2026-09-24 07:19:09 on dist-test-slave-zx46"
server_key: "694394066c374ced8b770f9080074384"
server_key_iv: "577747b9e01c896543a3419768203247"
server_key_version: "encryptionkey@0"
I20260924 07:19:52.034420 32299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:52.055403 32299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:52.058243 32299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:52.058444 32299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:52.065475 32299 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:37131
I20260924 07:19:52.065745 32366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:37131 every 8 connection(s)
I20260924 07:19:52.066557 32299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:19:52.070732 32367 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:19:52.072311 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap starting.
I20260924 07:19:52.072908 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 32299
I20260924 07:19:52.073386 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:40229
--local_ip_for_outbound_sockets=127.15.127.129
--tserver_master_addrs=127.15.127.190:37131
--webserver_port=34811
--webserver_interface=127.15.127.129
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:19:52.089551 32367 log.cc:824] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:52.100250 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:19:52.100615 32367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Bootstrap complete.
I20260924 07:19:52.104650 32367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:52.105166 32367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72810508f1ef4581a529f869a1098fbb, State: Initialized, Role: FOLLOWER
I20260924 07:19:52.105690 32367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:52.105834 32367 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:19:52.105924 32367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:19:52.106019 32367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:19:52.107792 32367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:52.108124 32367 leader_election.cc:302] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72810508f1ef4581a529f869a1098fbb; no voters:
I20260924 07:19:52.108454 32367 leader_election.cc:288] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:19:52.108533 32371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:19:52.108729 32371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [term 3 LEADER]: Becoming Leader. State: Replica: 72810508f1ef4581a529f869a1098fbb, State: Running, Role: LEADER
I20260924 07:19:52.109043 32371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } }
I20260924 07:19:52.109663 32367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:19:52.109935 32371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72810508f1ef4581a529f869a1098fbb. Latest consensus state: current_term: 3 leader_uuid: "72810508f1ef4581a529f869a1098fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } } }
I20260924 07:19:52.110055 32371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:52.110512 32372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "72810508f1ef4581a529f869a1098fbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72810508f1ef4581a529f869a1098fbb" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 37131 } } }
I20260924 07:19:52.110692 32372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb [sys.catalog]: This master's current role is: LEADER
I20260924 07:19:52.111558 32375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:19:52.117069 32375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:19:52.117839 32375 catalog_manager.cc:1259] Loaded cluster ID: 791312e0b861401cb76309251784bbc1
I20260924 07:19:52.117938 32375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:19:52.120318 32375 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:19:52.121093 32375 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Loaded TSK: 0
I20260924 07:19:52.121224 32375 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72810508f1ef4581a529f869a1098fbb: Loaded TSK: 1
F20260924 07:19:52.121991 32375 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbdcfe089ad google::LogMessage::Fail() at ??:0
@ 0x7fbdcfe0b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fbdcfe084de google::LogMessage::Flush() at ??:0
@ 0x7fbdcfe09a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbdd2acc18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbdd2ac4f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbdda76d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbdda74799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbdda748178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fbdda789df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fbdda7a882b std::function<>::operator()() at ??:0
@ 0x7fbdda747b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fbdda748f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbdda745579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fbdda789640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbdd205a3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbdd205ac83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fbdd205c3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fbdd204b512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbdd368f6db start_thread at ??:0
@ 0x7fbdce90661f clone at ??:0
W20260924 07:19:52.221335 32369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:52.221572 32369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:52.221616 32369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:52.225695 32369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:52.225745 32369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:52.225827 32369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:19:52.231067 32369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:40229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=34811
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:52.232097 32369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:52.233176 32369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:52.242702 32414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.242743 32415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.242920 32417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:52.242866 32369 server_base.cc:1034] running on GCE node
I20260924 07:19:52.243392 32369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:52.244041 32369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:52.245213 32369 hybrid_clock.cc:648] HybridClock initialized: now 1790234392245184 us; error 44 us; skew 500 ppm
Sep 24 07:19:52 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:52.248018 32369 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:19:52.249035 32369 webserver.cc:460] Webserver started at http://127.15.127.129:34811/ using document root <none> and password file <none>
I20260924 07:19:52.249603 32369 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:52.249665 32369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:52.253592 32369 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:19:52.255826 32424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.257226 32369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:19:52.257347 32369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "0effc2046fa3492da9ef20dd1674a47e"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "57279f1ee810f9757f06663662719694"
server_key_iv: "cc8b8609b56574ff696e5859858c8d05"
server_key_version: "encryptionkey@0"
I20260924 07:19:52.257901 32369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:52.283612 32369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:52.286861 32369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:52.287072 32369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:52.287734 32369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:52.288849 32431 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:19:52.292785 32369 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:19:52.292843 32369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.001s
I20260924 07:19:52.292914 32369 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:19:52.294570 32369 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:19:52.294625 32369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 07:19:52.294873 32431 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap starting.
I20260924 07:19:52.305500 32369 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:40229
I20260924 07:19:52.305733 32538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:40229 every 8 connection(s)
I20260924 07:19:52.306780 32369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:19:52.310427 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 32369
I20260924 07:19:52.310937 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:36879
--local_ip_for_outbound_sockets=127.15.127.130
--tserver_master_addrs=127.15.127.190:37131
--webserver_port=38573
--webserver_interface=127.15.127.130
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:19:52.316152 32431 log.cc:824] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:19:52.326263 32431 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:19:52.326699 32431 tablet_bootstrap.cc:492] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Bootstrap complete.
I20260924 07:19:52.328138 32431 ts_tablet_manager.cc:1397] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Time spent bootstrapping tablet: real 0.033s user 0.025s sys 0.004s
I20260924 07:19:52.331338 32431 raft_consensus.cc:348] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:52.331880 32431 raft_consensus.cc:729] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0effc2046fa3492da9ef20dd1674a47e, State: Initialized, Role: FOLLOWER
I20260924 07:19:52.332396 32431 consensus_queue.cc:260] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:52.333590 32431 ts_tablet_manager.cc:1428] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
W20260924 07:19:52.342002 32541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:19:52.309223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:33527)
0924 07:19:52.309824 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.309889 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:52.313447 (+ 3558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:52.340841 (+ 27394us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":160,"threads_started":1}
W20260924 07:19:52.342499 32539 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:37131 (0 consecutive failures): Network error: Failed to ping master at 127.15.127.190:37131: Client connection negotiation failed: client connection to 127.15.127.190:37131: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:19:52.343348 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.343049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:38787)
0924 07:19:52.343195 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.343283 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:19:52.344219 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.343927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:39469)
0924 07:19:52.344055 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.344137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:19:52.344427 32539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:19:52.431689 32542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:52.431945 32542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:52.431988 32542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:52.436089 32542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:52.436148 32542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:52.436204 32542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:19:52.441340 32542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:36879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=38573
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:52.442271 32542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:52.443189 32542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:52.452418 32550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.452395 32552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.452428 32549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:52.452979 32542 server_base.cc:1034] running on GCE node
I20260924 07:19:52.453444 32542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:52.454030 32542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:52.455211 32542 hybrid_clock.cc:648] HybridClock initialized: now 1790234392455180 us; error 44 us; skew 500 ppm
Sep 24 07:19:52 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:52.463546 32542 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:19:52.464505 32542 webserver.cc:460] Webserver started at http://127.15.127.130:38573/ using document root <none> and password file <none>
I20260924 07:19:52.465142 32542 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:52.465238 32542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:52.468979 32542 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:19:52.471122 32559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.472290 32542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:52.472448 32542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "02112738b7614f269a42eaf9085d29d2"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "7e87452e0df988060561f357aabc4701"
server_key_iv: "b618fd1b2142e9b6688408d85568bce8"
server_key_version: "encryptionkey@0"
I20260924 07:19:52.473007 32542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:52.503278 32542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:52.506326 32542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:52.506538 32542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:52.507122 32542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:52.508244 32566 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 07:19:52.512001 32566 ts_tablet_manager.cc:1852] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 07:19:52.512380 32566 ts_tablet_manager.cc:1905] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:19:52.516247 32566 ts_tablet_manager.cc:1918] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:19:52.516340 32566 log.cc:1196] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/ee813c97fd1c4c81a235505118c4b593
I20260924 07:19:52.516681 32566 ts_tablet_manager.cc:1939] T ee813c97fd1c4c81a235505118c4b593 P 02112738b7614f269a42eaf9085d29d2: Deleting consensus metadata
I20260924 07:19:52.517033 32542 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 07:19:52.517114 32542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20260924 07:19:52.517198 32542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:52.517247 32542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.526391 32542 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:36879
I20260924 07:19:52.526418 32673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:36879 every 8 connection(s)
I20260924 07:19:52.527227 32542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:19:52.530573 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.529324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:58237)
0924 07:19:52.529784 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.529901 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
W20260924 07:19:52.531026 32674 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:37131 (0 consecutive failures): Network error: Failed to ping master at 127.15.127.190:37131: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
I20260924 07:19:52.531435 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 32542
I20260924 07:19:52.531663 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.531426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:38983)
0924 07:19:52.531550 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.531609 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:19:52.531939 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:41031
--local_ip_for_outbound_sockets=127.15.127.131
--tserver_master_addrs=127.15.127.190:37131
--webserver_port=42985
--webserver_interface=127.15.127.131
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:19:52.532338 32676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.532079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:36591)
0924 07:19:52.532254 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.532292 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 07:19:52.532485 32674 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:19:52.643074 32677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:19:52.643368 32677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:19:52.643456 32677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:19:52.647629 32677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:19:52.647714 32677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:19:52.647802 32677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:19:52.653172 32677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:41031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=42985
--tserver_master_addrs=127.15.127.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:19:52.654234 32677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:19:52.655267 32677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:19:52.665084 32685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.665102 32682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:19:52.665103 32683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:19:52.665439 32677 server_base.cc:1034] running on GCE node
I20260924 07:19:52.665869 32677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:19:52.666455 32677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:19:52.667637 32677 hybrid_clock.cc:648] HybridClock initialized: now 1790234392667616 us; error 32 us; skew 500 ppm
Sep 24 07:19:52 dist-test-slave-zx46 krb5kdc[31224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:19:52.670437 32677 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:19:52.671267 32677 webserver.cc:460] Webserver started at http://127.15.127.131:42985/ using document root <none> and password file <none>
I20260924 07:19:52.671824 32677 fs_manager.cc:359] Metadata directory not provided
I20260924 07:19:52.671922 32677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:19:52.675648 32677 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:19:52.677812 32692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.679035 32677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:19:52.679163 32677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "9ba6fb22b5824c828e0e95cc71b36ac8"
format_stamp: "Formatted at 2026-09-24 07:19:10 on dist-test-slave-zx46"
server_key: "ac6b23f76857815d841337659baa8878"
server_key_iv: "989fb18bad96e9daa6fe0afea9b8497a"
server_key_version: "encryptionkey@0"
I20260924 07:19:52.679723 32677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:19:52.698434 32677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:19:52.701752 32677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:19:52.701961 32677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:19:52.702553 32677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:19:52.703352 32677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:19:52.703431 32677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.703503 32677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:19:52.703555 32677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:19:52.712407 32677 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:41031
I20260924 07:19:52.712476 337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:41031 every 8 connection(s)
I20260924 07:19:52.713467 32677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:19:52.717413 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.715566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:41659)
0924 07:19:52.716044 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.716156 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":105,"threads_started":1}
W20260924 07:19:52.717924 338 heartbeater.cc:644] Failed to heartbeat to 127.15.127.190:37131 (0 consecutive failures): Network error: Failed to ping master at 127.15.127.190:37131: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
I20260924 07:19:52.718477 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.718268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:49385)
0924 07:19:52.718380 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.718425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:19:52.718849 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 32677
I20260924 07:19:52.719089 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:52.718895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:40425)
0924 07:19:52.718994 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:52.719045 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:19:52.719238 338 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:19:53.345741 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.345173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:36501)
0924 07:19:53.345523 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:53.345594 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":81,"threads_started":1}
I20260924 07:19:53.533581 343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.533143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:59033)
0924 07:19:53.533422 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:53.533486 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":81,"threads_started":1}
I20260924 07:19:53.720134 344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.719738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:58499)
0924 07:19:53.719982 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:53.720038 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":65,"threads_started":1}
I20260924 07:19:53.806726 345 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:53.806949 345 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:53.808107 345 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
Sep 24 07:19:53 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.131@KRBTEST.COM
Sep 24 07:19:53 dist-test-slave-zx46 krb5kdc[31224](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234392, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.130@KRBTEST.COM
I20260924 07:19:53.825258 346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.808536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:40323 (local address 127.15.127.131:41031)
0924 07:19:53.808905 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:53.808929 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:53.808967 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:19:53.809936 (+ 969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:53.809945 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:53.810402 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:53.810739 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:53.812573 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.814156 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.815518 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.815983 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.816161 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:53.819632 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:53.819639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:53.819655 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:53.819690 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:53.821955 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:53.822736 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:53.822742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:53.822748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:53.822896 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:53.823319 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:53.823322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:53.823324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:53.823870 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:53.824002 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:53.824742 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 07:19:53.825062 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":112,"threads_started":1}
I20260924 07:19:53.825266 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.808349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:41031 (local address 127.15.127.129:40323)
0924 07:19:53.808478 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:53.808494 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:53.808604 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:53.810848 (+ 2244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:53.810854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:53.811487 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:53.812386 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:53.812406 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.814318 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.814322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:53.815388 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:53.815398 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.816110 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.816114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:53.816329 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:53.817055 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:53.817077 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:53.819397 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:53.822086 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:53.822093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:53.822115 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:53.822612 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:53.823004 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:53.823007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:53.823008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:53.823217 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:53.823998 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:53.824005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:53.824390 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 07:19:53.824610 (+ 220us) client_negotiation.cc:240] Negotiation successful
0924 07:19:53.825080 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":402,"spinlock_wait_cycles":71040}
I20260924 07:19:53.825428 347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.808649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:36879 (local address 127.15.127.129:38799)
0924 07:19:53.808945 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:53.808962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:19:53.809083 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:19:53.810848 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:19:53.810854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:19:53.811487 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:19:53.812385 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:53.812406 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.814342 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.814345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:53.815383 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:19:53.815393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.816093 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.816097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:19:53.816314 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:53.816984 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:19:53.817043 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:19:53.819394 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:19:53.822086 (+ 2692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:53.822093 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:53.822108 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:53.822612 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:53.823001 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:19:53.823004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:19:53.823006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:19:53.823217 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:19:53.823998 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:19:53.824005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:19:53.824386 (+ 381us) client_negotiation.cc:769] Sending connection context
0924 07:19:53.824610 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 07:19:53.825251 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":330,"spinlock_wait_cycles":32640,"thread_start_us":82,"threads_started":1}
I20260924 07:19:53.825634 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:53.808831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:38799 (local address 127.15.127.130:36879)
0924 07:19:53.809226 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 07:19:53.809238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:19:53.809277 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:19:53.810087 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:19:53.810094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:19:53.810402 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:19:53.810739 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:19:53.812573 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.814156 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.815509 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:19:53.815969 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:19:53.816158 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:19:53.819654 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:19:53.819659 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:19:53.819669 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:19:53.819696 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:19:53.821960 (+ 2264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:53.822736 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:53.822742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:53.822748 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:53.822904 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:19:53.823446 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:19:53.823449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:19:53.823451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:19:53.823883 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:19:53.824002 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 07:19:53.824742 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 07:19:53.825060 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":111,"threads_started":1}
W20260924 07:19:53.827921 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
W20260924 07:19:53.828099 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
I20260924 07:19:53.828143 32426 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0effc2046fa3492da9ef20dd1674a47e; no voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8
I20260924 07:19:53.828358 345 raft_consensus.cc:2726] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:19:54.347119 349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:54.346619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:39287)
0924 07:19:54.346944 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:54.347019 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":87,"threads_started":1}
I20260924 07:19:54.534739 350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:54.534340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:57183)
0924 07:19:54.534584 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:54.534637 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260924 07:19:54.721275 351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:54.720844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:46637)
0924 07:19:54.721131 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:54.721187 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":70,"threads_started":1}
I20260924 07:19:55.348306 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:55.347869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:34201)
0924 07:19:55.348148 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:55.348209 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260924 07:19:55.535924 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:55.535516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:51015)
0924 07:19:55.535772 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:55.535831 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":81,"threads_started":1}
I20260924 07:19:55.664805 354 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:55.664945 354 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:55.665411 354 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
W20260924 07:19:55.666307 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
W20260924 07:19:55.666500 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
I20260924 07:19:55.666564 32426 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0effc2046fa3492da9ef20dd1674a47e; no voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8
I20260924 07:19:55.666747 354 raft_consensus.cc:2726] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:19:55.722402 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:55.721987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:39359)
0924 07:19:55.722241 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:55.722296 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":87,"threads_started":1}
I20260924 07:19:56.349804 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:56.349394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:46343)
0924 07:19:56.349653 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:56.349709 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":95,"threads_started":1}
I20260924 07:19:56.537071 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:56.536615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:33597)
0924 07:19:56.536865 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:56.536917 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":94,"threads_started":1}
I20260924 07:19:56.723588 358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:56.723115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:51473)
0924 07:19:56.723443 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:56.723498 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":72,"threads_started":1}
I20260924 07:19:57.351130 359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:57.350689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:42615)
0924 07:19:57.350970 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:57.351036 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":71,"threads_started":1}
I20260924 07:19:57.538129 360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:57.537723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:34033)
0924 07:19:57.537986 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:57.538039 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":73,"threads_started":1}
I20260924 07:19:57.724641 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:57.724263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:56287)
0924 07:19:57.724506 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:57.724556 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260924 07:19:58.301752 362 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:19:58.301925 362 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:19:58.302323 362 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
W20260924 07:19:58.303172 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
W20260924 07:19:58.303289 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
I20260924 07:19:58.303339 32426 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0effc2046fa3492da9ef20dd1674a47e; no voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8
I20260924 07:19:58.303471 362 raft_consensus.cc:2726] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:19:58.352233 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:58.351833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:33843)
0924 07:19:58.352083 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:58.352139 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260924 07:19:58.539323 364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:58.538937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:59133)
0924 07:19:58.539180 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:58.539234 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260924 07:19:58.725965 365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:58.725568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:49725)
0924 07:19:58.725817 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:58.725874 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260924 07:19:59.353480 366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:59.353096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:38521)
0924 07:19:59.353334 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:59.353386 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":77,"threads_started":1}
I20260924 07:19:59.540503 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:59.540053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:45381)
0924 07:19:59.540350 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:59.540408 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":85,"threads_started":1}
I20260924 07:19:59.727267 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:19:59.726832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:35397)
0924 07:19:59.727120 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:19:59.727176 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":73,"threads_started":1}
I20260924 07:20:00.354640 369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.354235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:58953)
0924 07:20:00.354489 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.354544 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":88,"threads_started":1}
I20260924 07:20:00.541781 370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.541365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:57503)
0924 07:20:00.541612 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.541689 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":69,"threads_started":1}
I20260924 07:20:00.728471 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:00.728032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:43305)
0924 07:20:00.728318 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:00.728379 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260924 07:20:01.356029 372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:01.355625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:52949)
0924 07:20:01.355879 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:01.355937 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":75,"threads_started":1}
I20260924 07:20:01.543010 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:01.542577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:42493)
0924 07:20:01.542854 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:01.542916 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":91,"threads_started":1}
I20260924 07:20:01.729677 374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:01.729268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:41697)
0924 07:20:01.729525 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:01.729583 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260924 07:20:01.752837 375 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:01.752975 375 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:20:01.753446 375 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
W20260924 07:20:01.754354 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
W20260924 07:20:01.754482 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
I20260924 07:20:01.754519 32426 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0effc2046fa3492da9ef20dd1674a47e; no voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8
I20260924 07:20:01.754665 375 raft_consensus.cc:2726] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:20:02.357487 376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:02.357091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:38489)
0924 07:20:02.357339 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:02.357394 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":68,"threads_started":1}
I20260924 07:20:02.544436 377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:02.543929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:33393)
0924 07:20:02.544210 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:02.544330 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":95,"threads_started":1}
I20260924 07:20:02.730782 378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:02.730417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:38427)
0924 07:20:02.730641 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:02.730694 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":69,"threads_started":1}
I20260924 07:20:03.358615 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:03.358190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:36989)
0924 07:20:03.358467 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:03.358520 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":94,"threads_started":1}
I20260924 07:20:03.545709 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:03.545252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:40595)
0924 07:20:03.545529 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:03.545614 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260924 07:20:03.731979 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:03.731544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:33727)
0924 07:20:03.731823 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:03.731882 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260924 07:20:04.114619 382 raft_consensus.cc:482] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:04.114750 382 raft_consensus.cc:504] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ba6fb22b5824c828e0e95cc71b36ac8" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 41031 } } peers { permanent_uuid: "02112738b7614f269a42eaf9085d29d2" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 36879 } } peers { permanent_uuid: "0effc2046fa3492da9ef20dd1674a47e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40229 } }
I20260924 07:20:04.115095 382 leader_election.cc:288] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031), 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879)
W20260924 07:20:04.115911 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9ba6fb22b5824c828e0e95cc71b36ac8 (127.15.127.131:41031): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
W20260924 07:20:04.116065 32426 leader_election.cc:341] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 02112738b7614f269a42eaf9085d29d2 (127.15.127.130:36879): Not found: Tablet not found: ee813c97fd1c4c81a235505118c4b593
I20260924 07:20:04.116106 32426 leader_election.cc:302] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0effc2046fa3492da9ef20dd1674a47e; no voters: 02112738b7614f269a42eaf9085d29d2, 9ba6fb22b5824c828e0e95cc71b36ac8
I20260924 07:20:04.116238 382 raft_consensus.cc:2726] T ee813c97fd1c4c81a235505118c4b593 P 0effc2046fa3492da9ef20dd1674a47e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:20:04.359939 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:04.359513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:47297)
0924 07:20:04.359782 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:04.359837 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260924 07:20:04.546960 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:04.546498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:49091)
0924 07:20:04.546802 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:04.546860 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":83,"threads_started":1}
I20260924 07:20:04.733314 385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:04.732862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:45797)
0924 07:20:04.733153 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:04.733212 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":73,"threads_started":1}
I20260924 07:20:05.361258 386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:05.360748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:43355)
0924 07:20:05.361107 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:05.361161 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":107,"threads_started":1}
I20260924 07:20:05.548352 387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:05.547897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:43109)
0924 07:20:05.548190 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:05.548245 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":125,"threads_started":1}
I20260924 07:20:05.734555 388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:05.734065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:42215)
0924 07:20:05.734394 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:05.734457 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
I20260924 07:20:06.362370 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:06.361954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:42307)
0924 07:20:06.362207 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:06.362272 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":76,"threads_started":1}
I20260924 07:20:06.549732 390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:06.549324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:60485)
0924 07:20:06.549577 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:06.549637 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260924 07:20:06.735926 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:06.735507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.131:35861)
0924 07:20:06.735770 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:06.735833 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":80,"threads_started":1}
I20260924 07:20:07.363848 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:07.363464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.129:55885)
0924 07:20:07.363703 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:07.363758 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260924 07:20:07.550973 393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:07.550601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:37131 (local address 127.15.127.130:36477)
0924 07:20:07.550831 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:07.550880 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.190:37131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260924 07:20:07.722304 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 32369
I20260924 07:20:07.728034 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 32542
I20260924 07:20:07.733384 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 32677
2026-09-24T07:20:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59050 ms)
[----------] 30 tests from SecurityITest (244085 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:07 dist-test-slave-zx46 krb5kdc[396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:07 dist-test-slave-zx46 krb5kdc[396](info): set up 2 sockets
Sep 24 07:20:07 dist-test-slave-zx46 krb5kdc[396](info): commencing operation
krb5kdc: starting...
W20260924 07:20:08.780885 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:08 dist-test-slave-zx46 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:08.901528 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:38477
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:38477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:09.008172 409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:09.008476 409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:09.008558 409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:09.012610 409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:09.012696 409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:09.012734 409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:09.012792 409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:09.012863 409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:09.019950 409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:38477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:38477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:09.021479 409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:09.022867 409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:09.034283 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.034293 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.034293 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:09.034778 409 server_base.cc:1034] running on GCE node
I20260924 07:20:09.035282 409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:09.036746 409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:09.037912 409 hybrid_clock.cc:648] HybridClock initialized: now 1790234409037826 us; error 96 us; skew 500 ppm
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:09.041301 409 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:09.042079 409 webserver.cc:460] Webserver started at http://127.15.127.190:35097/ using document root <none> and password file <none>
I20260924 07:20:09.042591 409 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:09.042646 409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:09.042802 409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:09.044790 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "dbe2b316ce774ebf850a034bf832b8b0"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "4dba057275067ff17913fd3081e8c29d"
server_key_iv: "c91598fe9ba6019616c643cc3b129388"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.045378 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "dbe2b316ce774ebf850a034bf832b8b0"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "4dba057275067ff17913fd3081e8c29d"
server_key_iv: "c91598fe9ba6019616c643cc3b129388"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.048841 409 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:20:09.051159 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.052326 409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:09.052445 409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "dbe2b316ce774ebf850a034bf832b8b0"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "4dba057275067ff17913fd3081e8c29d"
server_key_iv: "c91598fe9ba6019616c643cc3b129388"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.052543 409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:09.067829 409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:09.070617 409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:09.070818 409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:09.078114 409 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:38477
I20260924 07:20:09.078125 476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:38477 every 8 connection(s)
I20260924 07:20:09.079084 409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:09.082151 477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:09.087535 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0: Bootstrap starting.
I20260924 07:20:09.087951 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 409
I20260924 07:20:09.088116 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:09.088428 15870 external_mini_cluster.cc:1428] Setting key 67902f585f2c55db5339d71aabc2e8b7
I20260924 07:20:09.090178 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:09.090891 477 log.cc:824] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:09.093102 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0: No bootstrap required, opened a new log
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:09.097445 477 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe2b316ce774ebf850a034bf832b8b0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38477 } }
I20260924 07:20:09.097688 477 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:09.097757 477 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe2b316ce774ebf850a034bf832b8b0, State: Initialized, Role: FOLLOWER
I20260924 07:20:09.098311 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe2b316ce774ebf850a034bf832b8b0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38477 } }
I20260924 07:20:09.098471 477 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:09.098541 477 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:09.098645 477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:09.099830 477 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe2b316ce774ebf850a034bf832b8b0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38477 } }
I20260924 07:20:09.100214 477 leader_election.cc:302] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe2b316ce774ebf850a034bf832b8b0; no voters:
I20260924 07:20:09.100545 477 leader_election.cc:288] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:09.100641 482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:09.100845 482 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [term 1 LEADER]: Becoming Leader. State: Replica: dbe2b316ce774ebf850a034bf832b8b0, State: Running, Role: LEADER
I20260924 07:20:09.101186 482 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe2b316ce774ebf850a034bf832b8b0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38477 } }
I20260924 07:20:09.101943 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:09.102778 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe2b316ce774ebf850a034bf832b8b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe2b316ce774ebf850a034bf832b8b0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38477 } } }
I20260924 07:20:09.102932 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:09.103364 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe2b316ce774ebf850a034bf832b8b0. Latest consensus state: current_term: 1 leader_uuid: "dbe2b316ce774ebf850a034bf832b8b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe2b316ce774ebf850a034bf832b8b0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 38477 } } }
I20260924 07:20:09.103459 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:09.104007 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:09.104197 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.089800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.15.127.190:38477)
0924 07:20:09.090198 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.090214 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.090249 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:09.090906 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.090922 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.091339 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.091696 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:09.092545 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.093675 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.094869 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.095328 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.095450 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.098344 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:09.098351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:09.098365 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:09.098394 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:09.100312 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.100885 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.100891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.100897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.101113 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.101469 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.101471 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.101473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.101933 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:09.102142 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.102761 (+ 619us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.103174 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":133,"threads_started":1}
I20260924 07:20:09.106817 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:09.111717 491 catalog_manager.cc:1347] Generated new cluster ID: db34595d011a49c0ac73327adfa1c2cc
I20260924 07:20:09.111773 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:09.120234 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:09.121394 491 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:09.127388 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0: Generated new TSK 0
I20260924 07:20:09.128027 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:20:09.159412 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38477
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:09.269552 503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:09.269851 503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:09.269894 503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:09.273772 503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:09.273820 503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:09.273874 503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:20:09.279126 503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:09.280054 503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:09.281028 503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:09.290758 508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.290776 511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.290769 509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:09.290817 503 server_base.cc:1034] running on GCE node
I20260924 07:20:09.291388 503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:09.291975 503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:09.293139 503 hybrid_clock.cc:648] HybridClock initialized: now 1790234409293123 us; error 28 us; skew 500 ppm
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:09.295856 503 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:20:09.296733 503 webserver.cc:460] Webserver started at http://127.15.127.129:41735/ using document root <none> and password file <none>
I20260924 07:20:09.297349 503 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:09.297438 503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:09.297663 503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:09.299829 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "34a497cabe374d4fa50e9c82c144d5ea"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "e553fbb4094ce8a093ec5befe8d6e4c1"
server_key_iv: "1d8eac1c940aed52abc8c3204957a1aa"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.300422 503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "34a497cabe374d4fa50e9c82c144d5ea"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "e553fbb4094ce8a093ec5befe8d6e4c1"
server_key_iv: "1d8eac1c940aed52abc8c3204957a1aa"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.304385 503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 07:20:09.306921 518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.308120 503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:09.308262 503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "34a497cabe374d4fa50e9c82c144d5ea"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "e553fbb4094ce8a093ec5befe8d6e4c1"
server_key_iv: "1d8eac1c940aed52abc8c3204957a1aa"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.308377 503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:09.320756 503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:09.323911 503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:09.324116 503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:09.324684 503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:09.325603 503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:09.325673 503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.325747 503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:09.325800 503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.334618 503 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:40485
I20260924 07:20:09.334636 631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:40485 every 8 connection(s)
I20260924 07:20:09.335583 503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:09.345525 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 503
I20260924 07:20:09.345675 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:09.345999 15870 external_mini_cluster.cc:1428] Setting key cf79d19e2366c28ab9c671c5c2fcceeb
I20260924 07:20:09.349366 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.337397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:38629 (local address 127.15.127.190:38477)
0924 07:20:09.337549 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.337553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.338354 (+ 801us) server_negotiation.cc:407] Connection header received
0924 07:20:09.339723 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.339727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.339786 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.339870 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:09.341063 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.341613 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.342784 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.343031 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.343150 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.345706 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:09.345709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:09.345712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:09.345765 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:09.347320 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.347830 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.347835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.347837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.347920 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.348255 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.348259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.348264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.348446 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:09.348547 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.349086 (+ 539us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.349208 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:20:09.350263 634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.337769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38477 (local address 127.15.127.129:38629)
0924 07:20:09.338198 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:09.338240 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:09.339491 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:09.340044 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:09.340051 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:09.340398 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:09.340880 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.340894 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.341758 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.341766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.342640 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.342649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.343193 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.343197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.343370 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.343924 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:09.343940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:09.345534 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:09.347442 (+ 1908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:09.347448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:09.347462 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:09.347726 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:09.348010 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:09.348015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:09.348017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:09.348160 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:09.348560 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:09.348568 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:09.348821 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 07:20:09.349007 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 07:20:09.349374 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260924 07:20:09.351531 632 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38477
I20260924 07:20:09.351783 632 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:09.352274 632 heartbeater.cc:511] Master 127.15.127.190:38477 requested a full tablet report, sending...
I20260924 07:20:09.354001 441 ts_manager.cc:194] Registered new tserver with Master: 34a497cabe374d4fa50e9c82c144d5ea (127.15.127.129:40485)
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:20:09.355684 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:38629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:20:09.372144 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38477
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:09.482944 637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:09.483192 637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:09.483234 637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:09.487015 637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:09.487056 637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:09.487109 637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:20:09.491973 637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:09.492864 637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:09.493767 637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:09.503327 645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.503333 642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.503444 643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:09.503592 637 server_base.cc:1034] running on GCE node
I20260924 07:20:09.504166 637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:09.504789 637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:09.505992 637 hybrid_clock.cc:648] HybridClock initialized: now 1790234409505962 us; error 51 us; skew 500 ppm
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:09.508818 637 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:20:09.509680 637 webserver.cc:460] Webserver started at http://127.15.127.130:44633/ using document root <none> and password file <none>
I20260924 07:20:09.510242 637 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:09.510318 637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:09.510535 637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:09.512630 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "7640596b47254e179f58899ae8232d0b"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "fd3a30e3dbef2fa20e11acf271e88361"
server_key_iv: "6908643e2f512c3b6b6ad45a706b6d03"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.513254 637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "7640596b47254e179f58899ae8232d0b"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "fd3a30e3dbef2fa20e11acf271e88361"
server_key_iv: "6908643e2f512c3b6b6ad45a706b6d03"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.516841 637 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:20:09.519163 652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.520342 637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:09.520479 637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "7640596b47254e179f58899ae8232d0b"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "fd3a30e3dbef2fa20e11acf271e88361"
server_key_iv: "6908643e2f512c3b6b6ad45a706b6d03"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.520586 637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:09.535404 637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:09.538738 637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:09.538940 637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:09.539511 637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:09.540315 637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:09.540385 637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.540452 637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:09.540500 637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.548882 637 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:35855
I20260924 07:20:09.548949 765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:35855 every 8 connection(s)
I20260924 07:20:09.549836 637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:09.558413 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 637
I20260924 07:20:09.558555 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:09.558947 15870 external_mini_cluster.cc:1428] Setting key d7101ac9f1c50588243b86d85bc2a94b
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:09.563691 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.551683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:41529 (local address 127.15.127.190:38477)
0924 07:20:09.551816 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.551819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.552622 (+ 803us) server_negotiation.cc:407] Connection header received
0924 07:20:09.554003 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.554006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.554048 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.554117 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:09.555220 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.555739 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.556880 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.557200 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.557288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.559970 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:09.559976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:09.559983 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:09.560006 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:09.561581 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.562183 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.562186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.562187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.562269 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.562648 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.562650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.562651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.562824 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:09.562897 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.563488 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.563579 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:20:09.564941 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.552061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38477 (local address 127.15.127.130:41529)
0924 07:20:09.552474 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:09.552514 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:09.553801 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:09.554275 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:09.554281 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:09.554629 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:09.555089 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.555103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.555867 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.555871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.556783 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.556791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.557334 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.557337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.557480 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.558017 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:09.558033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:09.559809 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:09.561700 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:09.561706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:09.561721 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:09.562079 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:09.562363 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:09.562368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:09.562371 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:09.562553 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:09.562932 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:09.562939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:09.563253 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 07:20:09.563458 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 07:20:09.563809 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":97,"threads_started":1}
I20260924 07:20:09.566191 766 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38477
I20260924 07:20:09.566458 766 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:09.566954 766 heartbeater.cc:511] Master 127.15.127.190:38477 requested a full tablet report, sending...
I20260924 07:20:09.568053 441 ts_manager.cc:194] Registered new tserver with Master: 7640596b47254e179f58899ae8232d0b (127.15.127.130:35855)
I20260924 07:20:09.568614 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:41529
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:20:09.587726 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38477
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:09.695889 771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:09.696168 771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:09.696245 771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:09.700384 771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:09.700465 771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:09.700557 771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:20:09.705942 771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:38477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:09.706929 771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:09.707912 771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:09.717175 776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:09.717175 779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:09.717262 771 server_base.cc:1034] running on GCE node
W20260924 07:20:09.717367 777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:09.717732 771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:09.718360 771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:09.719539 771 hybrid_clock.cc:648] HybridClock initialized: now 1790234409719513 us; error 51 us; skew 500 ppm
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:09.722239 771 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:20:09.723134 771 webserver.cc:460] Webserver started at http://127.15.127.131:45151/ using document root <none> and password file <none>
I20260924 07:20:09.723811 771 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:09.723904 771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:09.724116 771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:09.726317 771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "ab9894c8266c43dab5bb2282232740a5"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "d3b83b8241052cb4db1459a475605b26"
server_key_iv: "87389df78209d24f1759bd2654db0c85"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.726897 771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "ab9894c8266c43dab5bb2282232740a5"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "d3b83b8241052cb4db1459a475605b26"
server_key_iv: "87389df78209d24f1759bd2654db0c85"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.730511 771 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:09.732929 786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.734166 771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:09.734306 771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "ab9894c8266c43dab5bb2282232740a5"
format_stamp: "Formatted at 2026-09-24 07:20:09 on dist-test-slave-zx46"
server_key: "d3b83b8241052cb4db1459a475605b26"
server_key_iv: "87389df78209d24f1759bd2654db0c85"
server_key_version: "encryptionkey@0"
I20260924 07:20:09.734447 771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:09.761875 771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:09.764776 771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:09.764997 771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:09.765609 771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:09.766417 771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:09.766486 771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.766556 771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:09.766606 771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:09.775223 771 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:36457
I20260924 07:20:09.775250 899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:36457 every 8 connection(s)
I20260924 07:20:09.776109 771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:09.783986 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 771
I20260924 07:20:09.784137 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:09.784442 15870 external_mini_cluster.cc:1428] Setting key f99211a86b2f069ef13e738e5f4a710c
Sep 24 07:20:09 dist-test-slave-zx46 krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234409, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:09.789377 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.777938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:36245 (local address 127.15.127.190:38477)
0924 07:20:09.778085 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.778090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.778926 (+ 836us) server_negotiation.cc:407] Connection header received
0924 07:20:09.780197 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.780200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.780251 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.780327 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:09.781493 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.782013 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.783052 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.783245 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.783336 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.786264 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:09.786267 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:09.786269 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:09.786291 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:09.787629 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.788085 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.788087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.788088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.788161 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.788447 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.788449 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.788450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.788610 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:09.788683 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.789172 (+ 489us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.789262 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:20:09.790172 902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.778360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:38477 (local address 127.15.127.131:36245)
0924 07:20:09.778779 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:09.778820 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:09.780014 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:09.780485 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:09.780492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:09.780817 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:09.781372 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.781386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.782113 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.782115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.782954 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.782962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.783379 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.783382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.783513 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.784134 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:09.784159 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:09.786112 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:09.787728 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:09.787732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:09.787751 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:09.787999 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:09.788239 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:09.788242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:09.788244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:09.788371 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:09.788693 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:09.788697 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:09.788913 (+ 216us) client_negotiation.cc:769] Sending connection context
0924 07:20:09.789114 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 07:20:09.789355 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260924 07:20:09.791179 900 heartbeater.cc:347] Connected to a master server at 127.15.127.190:38477
I20260924 07:20:09.791380 900 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:09.791791 900 heartbeater.cc:511] Master 127.15.127.190:38477 requested a full tablet report, sending...
I20260924 07:20:09.792683 441 ts_manager.cc:194] Registered new tserver with Master: ab9894c8266c43dab5bb2282232740a5 (127.15.127.131:36457)
I20260924 07:20:09.793308 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:36245
I20260924 07:20:09.798273 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:09.807152 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.800150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56912 (local address 127.15.127.190:38477)
0924 07:20:09.800262 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.800267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.800425 (+ 158us) server_negotiation.cc:407] Connection header received
0924 07:20:09.800517 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.800519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.800554 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.800618 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:09.801361 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.801846 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.802669 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.802883 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.802959 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.804325 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:09.804328 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:09.804330 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:09.804349 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:09.805611 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.805977 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.805980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.805981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.806052 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:09.806270 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:09.806272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:09.806273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:09.806485 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:09.806629 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.806879 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.806990 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:20:09.809731 441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:09.812783 441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:09.827483 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.821617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.15.127.131:36457)
0924 07:20:09.822000 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.822011 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.822329 (+ 318us) server_negotiation.cc:407] Connection header received
0924 07:20:09.822411 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.822415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.822550 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.822892 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:09.823305 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.824154 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.825929 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.826813 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.826942 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.827086 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.827246 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.827340 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":175,"threads_started":1}
I20260924 07:20:09.827512 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.820791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:35855 (local address 127.0.0.1:38378)
0924 07:20:09.821558 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:09.821606 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:09.821754 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:09.822720 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:09.822726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:09.822768 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:09.822979 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.822987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.824535 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.824538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.825985 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.825996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.827030 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.827033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.827121 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.827137 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 07:20:09.827205 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:20:09.827340 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":183,"spinlock_wait_cycles":5248,"thread_start_us":75,"threads_started":1}
I20260924 07:20:09.827519 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.820961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:36457 (local address 127.0.0.1:48754)
0924 07:20:09.821571 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:09.821615 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:09.821743 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:09.822720 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:09.822726 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:09.822782 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:09.823189 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.823195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.824283 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.824287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.825796 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.825804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.826925 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.826928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.827048 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.827090 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:20:09.827205 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 07:20:09.827340 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":288,"spinlock_wait_cycles":44672,"thread_start_us":113,"threads_started":1}
I20260924 07:20:09.827965 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.820924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.15.127.130:35855)
0924 07:20:09.822228 (+ 1304us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.822239 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.822272 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:20:09.822348 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.822355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.822548 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.823127 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:09.823282 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.824392 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.826136 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.826929 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.827039 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.827134 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.827715 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.827830 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1206,"thread_start_us":138,"threads_started":1}
I20260924 07:20:09.828763 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.821214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40485 (local address 127.0.0.1:55144)
0924 07:20:09.821987 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:09.821999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:09.822080 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:09.822734 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:09.822737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:09.822795 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:09.822998 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.823005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.824560 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.824564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.825986 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:09.825996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.828385 (+ 2389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.828389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:09.828474 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.828497 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:20:09.828575 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:20:09.828638 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":191,"spinlock_wait_cycles":80768,"thread_start_us":92,"threads_started":1}
I20260924 07:20:09.830906 700 tablet_service.cc:1467] Processing CreateTablet for tablet 938b755342524d3eae02e7cbf6e65ee4 (DEFAULT_TABLE table=test-table [id=6425e67a84a246ed9f4ba22eb5f89cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:09.830907 834 tablet_service.cc:1467] Processing CreateTablet for tablet 938b755342524d3eae02e7cbf6e65ee4 (DEFAULT_TABLE table=test-table [id=6425e67a84a246ed9f4ba22eb5f89cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:09.828874 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:09.821601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.15.127.129:40485)
0924 07:20:09.822025 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:09.822040 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:09.822065 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:20:09.822216 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:09.822221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:09.822441 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:09.822596 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:09.823126 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.824392 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.827195 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:09.828043 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:09.828166 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:09.828319 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:09.828629 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 07:20:09.828733 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":73,"threads_started":1}
I20260924 07:20:09.832049 834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938b755342524d3eae02e7cbf6e65ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:09.832048 700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938b755342524d3eae02e7cbf6e65ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:09.834180 566 tablet_service.cc:1467] Processing CreateTablet for tablet 938b755342524d3eae02e7cbf6e65ee4 (DEFAULT_TABLE table=test-table [id=6425e67a84a246ed9f4ba22eb5f89cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:09.835047 566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938b755342524d3eae02e7cbf6e65ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:09.839395 918 tablet_bootstrap.cc:492] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Bootstrap starting.
I20260924 07:20:09.840013 917 tablet_bootstrap.cc:492] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Bootstrap starting.
I20260924 07:20:09.841502 919 tablet_bootstrap.cc:492] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea: Bootstrap starting.
I20260924 07:20:09.842371 917 tablet_bootstrap.cc:654] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:09.843108 918 tablet_bootstrap.cc:654] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:09.843516 917 log.cc:824] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:09.843611 919 tablet_bootstrap.cc:654] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:09.844439 918 log.cc:824] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:09.844913 919 log.cc:824] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:09.845541 917 tablet_bootstrap.cc:492] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: No bootstrap required, opened a new log
I20260924 07:20:09.845737 917 ts_tablet_manager.cc:1397] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260924 07:20:09.846901 919 tablet_bootstrap.cc:492] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea: No bootstrap required, opened a new log
I20260924 07:20:09.847080 919 ts_tablet_manager.cc:1397] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 07:20:09.847132 918 tablet_bootstrap.cc:492] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: No bootstrap required, opened a new log
I20260924 07:20:09.847301 918 ts_tablet_manager.cc:1397] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260924 07:20:09.848758 917 raft_consensus.cc:348] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:09.848990 917 raft_consensus.cc:374] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:09.849094 917 raft_consensus.cc:729] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7640596b47254e179f58899ae8232d0b, State: Initialized, Role: FOLLOWER
I20260924 07:20:09.849601 917 consensus_queue.cc:260] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:09.850214 766 heartbeater.cc:503] Master 127.15.127.190:38477 was elected leader, sending a full tablet report...
I20260924 07:20:09.850559 917 ts_tablet_manager.cc:1428] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:20:09.850993 918 raft_consensus.cc:348] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:09.850993 919 raft_consensus.cc:348] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:09.851226 919 raft_consensus.cc:374] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:09.851226 918 raft_consensus.cc:374] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:09.851292 918 raft_consensus.cc:729] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9894c8266c43dab5bb2282232740a5, State: Initialized, Role: FOLLOWER
I20260924 07:20:09.851357 919 raft_consensus.cc:729] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34a497cabe374d4fa50e9c82c144d5ea, State: Initialized, Role: FOLLOWER
I20260924 07:20:09.851768 919 consensus_queue.cc:260] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:09.851764 918 consensus_queue.cc:260] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:09.852672 919 ts_tablet_manager.cc:1428] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 07:20:09.853142 918 ts_tablet_manager.cc:1428] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:20:09.853206 632 heartbeater.cc:503] Master 127.15.127.190:38477 was elected leader, sending a full tablet report...
I20260924 07:20:09.853458 900 heartbeater.cc:503] Master 127.15.127.190:38477 was elected leader, sending a full tablet report...
W20260924 07:20:10.027163 901 tablet.cc:2369] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:10.031416 923 raft_consensus.cc:482] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:10.031625 923 raft_consensus.cc:504] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:10.032862 923 leader_election.cc:288] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9894c8266c43dab5bb2282232740a5 (127.15.127.131:36457), 34a497cabe374d4fa50e9c82c144d5ea (127.15.127.129:40485)
I20260924 07:20:10.037670 916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.033150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:34137 (local address 127.15.127.131:36457)
0924 07:20:10.033292 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:10.033297 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:10.033315 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:20:10.033506 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:10.033510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:10.033564 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:10.033655 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:10.034011 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.034783 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.036412 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.036986 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.037125 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.037179 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:10.037493 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:20:10.037558 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:20:10.037670 915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.033429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:39115 (local address 127.15.127.129:40485)
0924 07:20:10.033560 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:10.033565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:10.033645 (+ 80us) server_negotiation.cc:407] Connection header received
0924 07:20:10.033797 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:10.033800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:10.033855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:10.033914 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:10.034185 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.034869 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.036412 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.036986 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.037146 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.037188 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:10.037466 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 07:20:10.037543 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:20:10.037704 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.033060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:36457 (local address 127.15.127.130:34137)
0924 07:20:10.033255 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:10.033269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:10.033362 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:10.033684 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:10.033688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:10.033702 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:10.033861 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.033867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.034884 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.034886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.036291 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.036301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.037114 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.037117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.037248 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.037361 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 07:20:10.037435 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:20:10.037545 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":162}
I20260924 07:20:10.037715 926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.033235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40485 (local address 127.15.127.130:39115)
0924 07:20:10.033539 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:10.033556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:10.033684 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:10.033952 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:10.033954 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:10.033987 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:10.034091 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.034095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.034956 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.034959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.036292 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.036302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.037114 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.037117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.037288 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.037360 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 07:20:10.037435 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:20:10.037543 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":143,"thread_start_us":125,"threads_started":1}
I20260924 07:20:10.038412 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938b755342524d3eae02e7cbf6e65ee4" candidate_uuid: "7640596b47254e179f58899ae8232d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9894c8266c43dab5bb2282232740a5" is_pre_election: true
I20260924 07:20:10.038561 586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938b755342524d3eae02e7cbf6e65ee4" candidate_uuid: "7640596b47254e179f58899ae8232d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a497cabe374d4fa50e9c82c144d5ea" is_pre_election: true
I20260924 07:20:10.038718 854 raft_consensus.cc:2454] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7640596b47254e179f58899ae8232d0b in term 0.
I20260924 07:20:10.038815 586 raft_consensus.cc:2454] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7640596b47254e179f58899ae8232d0b in term 0.
I20260924 07:20:10.039250 654 leader_election.cc:302] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 34a497cabe374d4fa50e9c82c144d5ea, 7640596b47254e179f58899ae8232d0b, ab9894c8266c43dab5bb2282232740a5; no voters:
I20260924 07:20:10.039532 923 raft_consensus.cc:2781] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:10.039635 923 raft_consensus.cc:482] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:10.039692 923 raft_consensus.cc:3037] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:10.040740 923 raft_consensus.cc:504] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:10.041146 923 leader_election.cc:288] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [CANDIDATE]: Term 1 election: Requested vote from peers ab9894c8266c43dab5bb2282232740a5 (127.15.127.131:36457), 34a497cabe374d4fa50e9c82c144d5ea (127.15.127.129:40485)
I20260924 07:20:10.041443 854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938b755342524d3eae02e7cbf6e65ee4" candidate_uuid: "7640596b47254e179f58899ae8232d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9894c8266c43dab5bb2282232740a5"
I20260924 07:20:10.041443 586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938b755342524d3eae02e7cbf6e65ee4" candidate_uuid: "7640596b47254e179f58899ae8232d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "34a497cabe374d4fa50e9c82c144d5ea"
I20260924 07:20:10.041574 586 raft_consensus.cc:3037] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:10.041574 854 raft_consensus.cc:3037] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:10.042804 854 raft_consensus.cc:2454] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7640596b47254e179f58899ae8232d0b in term 1.
I20260924 07:20:10.042804 586 raft_consensus.cc:2454] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7640596b47254e179f58899ae8232d0b in term 1.
I20260924 07:20:10.043097 653 leader_election.cc:302] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 34a497cabe374d4fa50e9c82c144d5ea, 7640596b47254e179f58899ae8232d0b, ab9894c8266c43dab5bb2282232740a5; no voters:
I20260924 07:20:10.043347 923 raft_consensus.cc:2781] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:10.043542 923 raft_consensus.cc:686] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 1 LEADER]: Becoming Leader. State: Replica: 7640596b47254e179f58899ae8232d0b, State: Running, Role: LEADER
I20260924 07:20:10.043859 923 consensus_queue.cc:237] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } }
I20260924 07:20:10.047446 441 catalog_manager.cc:5495] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 7640596b47254e179f58899ae8232d0b (127.15.127.130). New cstate: current_term: 1 leader_uuid: "7640596b47254e179f58899ae8232d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7640596b47254e179f58899ae8232d0b" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 } health_report { overall_health: UNKNOWN } } }
W20260924 07:20:10.050961 767 tablet.cc:2369] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:10.063114 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.060004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.15.127.130:35855)
0924 07:20:10.060114 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:10.060118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:10.060132 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:20:10.060283 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:10.060285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:10.060335 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:10.060421 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:10.060630 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.061168 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.061944 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.062139 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.062209 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.062443 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:10.062813 (+ 370us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:10.062899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:10.062955 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 07:20:10.063019 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:20:10.068478 586 raft_consensus.cc:1261] T 938b755342524d3eae02e7cbf6e65ee4 P 34a497cabe374d4fa50e9c82c144d5ea [term 1 FOLLOWER]: Refusing update from remote peer 7640596b47254e179f58899ae8232d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:10.068499 854 raft_consensus.cc:1261] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 1 FOLLOWER]: Refusing update from remote peer 7640596b47254e179f58899ae8232d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:10.069191 923 consensus_queue.cc:1035] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9894c8266c43dab5bb2282232740a5" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:10.069415 927 consensus_queue.cc:1035] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "34a497cabe374d4fa50e9c82c144d5ea" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 40485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:10.080777 931 mvcc.cc:205] Tried to move back new op lower bound from 7332800143633960960 to 7332800143543033856. Current Snapshot: MvccSnapshot[applied={T|T < 7332800143633960960}]
I20260924 07:20:10.084160 441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:10.084316 441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:10.086525 441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe2b316ce774ebf850a034bf832b8b0: Sending DeleteTablet for 3 replicas of tablet 938b755342524d3eae02e7cbf6e65ee4
I20260924 07:20:10.087198 700 tablet_service.cc:1514] Processing DeleteTablet for tablet 938b755342524d3eae02e7cbf6e65ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:38378
I20260924 07:20:10.087353 566 tablet_service.cc:1514] Processing DeleteTablet for tablet 938b755342524d3eae02e7cbf6e65ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:55144
I20260924 07:20:10.087667 939 tablet_replica.cc:330] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: stopping tablet replica
I20260924 07:20:10.088023 939 raft_consensus.cc:2229] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:10.087986 834 tablet_service.cc:1514] Processing DeleteTablet for tablet 938b755342524d3eae02e7cbf6e65ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:10 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:48754
I20260924 07:20:10.088444 939 raft_consensus.cc:2258] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:10.089049 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 503
I20260924 07:20:10.090840 941 tablet_replica.cc:330] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: stopping tablet replica
I20260924 07:20:10.091140 941 raft_consensus.cc:2229] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:10.091430 941 raft_consensus.cc:2258] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:10.093009 941 ts_tablet_manager.cc:1905] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:10.093009 939 ts_tablet_manager.cc:1905] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:10.095937 941 ts_tablet_manager.cc:1918] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:10.095947 939 ts_tablet_manager.cc:1918] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:10.096043 941 log.cc:1196] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/938b755342524d3eae02e7cbf6e65ee4
I20260924 07:20:10.096153 939 log.cc:1196] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/938b755342524d3eae02e7cbf6e65ee4
I20260924 07:20:10.096413 941 ts_tablet_manager.cc:1939] T 938b755342524d3eae02e7cbf6e65ee4 P ab9894c8266c43dab5bb2282232740a5: Deleting consensus metadata
I20260924 07:20:10.096635 939 ts_tablet_manager.cc:1939] T 938b755342524d3eae02e7cbf6e65ee4 P 7640596b47254e179f58899ae8232d0b: Deleting consensus metadata
I20260924 07:20:10.097661 428 catalog_manager.cc:4841] TS 7640596b47254e179f58899ae8232d0b (127.15.127.130:35855): tablet 938b755342524d3eae02e7cbf6e65ee4 (table test-table [id=6425e67a84a246ed9f4ba22eb5f89cf7]) successfully deleted
I20260924 07:20:10.097661 425 catalog_manager.cc:4841] TS ab9894c8266c43dab5bb2282232740a5 (127.15.127.131:36457): tablet 938b755342524d3eae02e7cbf6e65ee4 (table test-table [id=6425e67a84a246ed9f4ba22eb5f89cf7]) successfully deleted
W20260924 07:20:10.101846 426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:40485 (error 108)
I20260924 07:20:10.101917 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 637
I20260924 07:20:10.103302 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.103063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:40485 (local address 127.0.0.1:55146)
0924 07:20:10.103177 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:10.103236 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:40485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 07:20:10.103487 426 catalog_manager.cc:4584] TS 34a497cabe374d4fa50e9c82c144d5ea (127.15.127.129:40485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 938b755342524d3eae02e7cbf6e65ee4: Network error: Client connection negotiation failed: client connection to 127.15.127.129:40485: connect: Connection refused (error 111)
I20260924 07:20:10.108919 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 771
I20260924 07:20:10.114678 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 409
2026-09-24T07:20:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2381 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T07:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:10Z Disabled control of system clock
I20260924 07:20:10.144250 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:35859
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:35859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:20:10.250694 946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:10.250943 946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:10.250977 946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:10.254976 946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:10.255030 946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:10.255045 946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:10.255062 946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:10.255079 946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:10.260213 946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:35859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:35859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:10.261293 946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:10.262599 946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:10.271113 952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.271106 951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.271173 954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:10.271266 946 server_base.cc:1034] running on GCE node
I20260924 07:20:10.271977 946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:10.273051 946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:10.274222 946 hybrid_clock.cc:648] HybridClock initialized: now 1790234410274189 us; error 46 us; skew 500 ppm
I20260924 07:20:10.276012 946 webserver.cc:460] Webserver started at http://127.15.127.190:36153/ using document root <none> and password file <none>
I20260924 07:20:10.276576 946 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:10.276638 946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:10.276860 946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:10.278959 946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "e029b82a9ba244d4a875aca2d85d7b8e"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "33d460488d5573e7b43388be405c79c2"
server_key_iv: "9a0441cb4c021d8583369e4f8299ffeb"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.279513 946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "e029b82a9ba244d4a875aca2d85d7b8e"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "33d460488d5573e7b43388be405c79c2"
server_key_iv: "9a0441cb4c021d8583369e4f8299ffeb"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.283208 946 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 07:20:10.285605 960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.286806 946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:10.286962 946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "e029b82a9ba244d4a875aca2d85d7b8e"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "33d460488d5573e7b43388be405c79c2"
server_key_iv: "9a0441cb4c021d8583369e4f8299ffeb"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.287072 946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:10.310899 946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:10.311604 946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:10.311815 946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:10.318988 946 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:35859
I20260924 07:20:10.318993 1012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:35859 every 8 connection(s)
I20260924 07:20:10.319922 946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:10.323012 1013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:10.328219 1013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e: Bootstrap starting.
I20260924 07:20:10.329614 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 946
I20260924 07:20:10.329746 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:10.330031 15870 external_mini_cluster.cc:1428] Setting key 19fe4a62a77f59cd9e19a2946a7653e8
I20260924 07:20:10.331043 1013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:10.331864 1013 log.cc:824] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:10.334046 1013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e: No bootstrap required, opened a new log
I20260924 07:20:10.338313 1013 raft_consensus.cc:348] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35859 } }
I20260924 07:20:10.338577 1013 raft_consensus.cc:374] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:10.338662 1013 raft_consensus.cc:729] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e029b82a9ba244d4a875aca2d85d7b8e, State: Initialized, Role: FOLLOWER
I20260924 07:20:10.339118 1013 consensus_queue.cc:260] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35859 } }
I20260924 07:20:10.339272 1013 raft_consensus.cc:388] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:10.339359 1013 raft_consensus.cc:482] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:10.339407 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.331174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48384 (local address 127.15.127.190:35859)
0924 07:20:10.331788 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:10.331802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:10.331837 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:10.332648 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:10.332674 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:10.333105 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:10.333388 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:10.333708 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.334721 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.335935 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.336417 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.336531 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.337481 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:10.337486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:10.337500 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:10.337894 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:10.337951 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:10.337970 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:10.338045 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:10.338216 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 07:20:10.338516 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260924 07:20:10.339457 1013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:10.340544 1013 raft_consensus.cc:504] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35859 } }
I20260924 07:20:10.340885 1013 leader_election.cc:302] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e029b82a9ba244d4a875aca2d85d7b8e; no voters:
I20260924 07:20:10.341229 1013 leader_election.cc:288] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:10.341297 1018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:10.341497 1018 raft_consensus.cc:686] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [term 1 LEADER]: Becoming Leader. State: Replica: e029b82a9ba244d4a875aca2d85d7b8e, State: Running, Role: LEADER
I20260924 07:20:10.342077 1018 consensus_queue.cc:237] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35859 } }
I20260924 07:20:10.342501 1013 sys_catalog.cc:564] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:10.343740 1020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35859 } } }
I20260924 07:20:10.343787 1019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e029b82a9ba244d4a875aca2d85d7b8e. Latest consensus state: current_term: 1 leader_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e029b82a9ba244d4a875aca2d85d7b8e" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35859 } } }
I20260924 07:20:10.343884 1020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:10.343890 1019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:10.344334 1027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:10.347467 1027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:10.352901 1027 catalog_manager.cc:1347] Generated new cluster ID: 5b3a25f3db00430fb8b5bfffa2cd9ccf
I20260924 07:20:10.352977 1027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:10.374228 1027 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:10.375196 1027 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:10.387094 1027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e: Generated new TSK 0
I20260924 07:20:10.387997 1027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:10.399871 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35859
--builtin_ntp_servers=127.15.127.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:20:10.513424 1037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:10.513655 1037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:10.513691 1037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:10.517926 1037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:10.517974 1037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:10.518055 1037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:20:10.522675 1037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:10.523761 1037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:10.525308 1037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:10.534680 1043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.534713 1042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.534765 1045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:10.535032 1037 server_base.cc:1034] running on GCE node
I20260924 07:20:10.535418 1037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:10.536020 1037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:10.537199 1037 hybrid_clock.cc:648] HybridClock initialized: now 1790234410537183 us; error 24 us; skew 500 ppm
I20260924 07:20:10.539001 1037 webserver.cc:460] Webserver started at http://127.15.127.129:39939/ using document root <none> and password file <none>
I20260924 07:20:10.539559 1037 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:10.539623 1037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:10.539891 1037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:10.542057 1037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "60fb01800a0e4d01b55dafd1c40d844c"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "d0a2ce8d828592e16435bb9fa3f5fc7e"
server_key_iv: "110eb1d75a72bc20768bfb1cc3a060a3"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.542634 1037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "60fb01800a0e4d01b55dafd1c40d844c"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "d0a2ce8d828592e16435bb9fa3f5fc7e"
server_key_iv: "110eb1d75a72bc20768bfb1cc3a060a3"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.546521 1037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:20:10.548921 1051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.550195 1037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:10.550295 1037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "60fb01800a0e4d01b55dafd1c40d844c"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "d0a2ce8d828592e16435bb9fa3f5fc7e"
server_key_iv: "110eb1d75a72bc20768bfb1cc3a060a3"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.550428 1037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:10.594964 1037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:10.595775 1037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:10.595948 1037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:10.596550 1037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:10.597573 1037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:10.597620 1037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.597661 1037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:10.597676 1037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.606555 1037 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:43931
I20260924 07:20:10.606626 1164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:43931 every 8 connection(s)
I20260924 07:20:10.607506 1037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:10.616674 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1037
I20260924 07:20:10.616838 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:10.617187 15870 external_mini_cluster.cc:1428] Setting key fa88e4a7a8afb8cb4e1f91b589dfd654
I20260924 07:20:10.617467 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.609498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:38895 (local address 127.15.127.190:35859)
0924 07:20:10.609697 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:10.609701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:10.610808 (+ 1107us) server_negotiation.cc:407] Connection header received
0924 07:20:10.612322 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:10.612325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:10.612391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:10.612502 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:10.613518 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.614067 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.615248 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.615479 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.615594 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.616878 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:10.616882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:10.616884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:10.616917 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:10.616941 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:10.616953 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:10.617085 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:10.617261 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 07:20:10.617324 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 07:20:10.618575 1167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.609930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35859 (local address 127.15.127.129:38895)
0924 07:20:10.610630 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:10.610678 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:10.612073 (+ 1395us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:10.612694 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:10.612702 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:10.612794 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:10.613370 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.613383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.614206 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.614209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.615107 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.615116 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.615649 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.615654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.615815 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.616584 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:10.616585 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:10.616628 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:10.616712 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:10.617086 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:10.617098 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:10.617107 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:20:10.617261 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:20:10.617558 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":119,"threads_started":1}
I20260924 07:20:10.619290 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35859
--builtin_ntp_servers=127.15.127.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:20:10.619899 1165 heartbeater.cc:347] Connected to a master server at 127.15.127.190:35859
I20260924 07:20:10.620150 1165 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:10.620790 1165 heartbeater.cc:511] Master 127.15.127.190:35859 requested a full tablet report, sending...
I20260924 07:20:10.622391 977 ts_manager.cc:194] Registered new tserver with Master: 60fb01800a0e4d01b55dafd1c40d844c (127.15.127.129:43931)
I20260924 07:20:10.623600 977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.129:38895
W20260924 07:20:10.729494 1168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:10.729809 1168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:10.729882 1168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:10.733897 1168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:10.733990 1168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:10.734103 1168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:20:10.739043 1168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:10.740185 1168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:10.741663 1168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:10.751405 1173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.751466 1176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.751405 1174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:10.751619 1168 server_base.cc:1034] running on GCE node
I20260924 07:20:10.752107 1168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:10.752758 1168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:10.753942 1168 hybrid_clock.cc:648] HybridClock initialized: now 1790234410753898 us; error 52 us; skew 500 ppm
I20260924 07:20:10.755863 1168 webserver.cc:460] Webserver started at http://127.15.127.130:33303/ using document root <none> and password file <none>
I20260924 07:20:10.756439 1168 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:10.756502 1168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:10.756744 1168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:10.758817 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "543357c1819f4a98bbdf062ba53813f3"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "32ebd15a3d8a378311ebf9b6b358dd33"
server_key_iv: "c6fbe04d089ade73cec43158b67b7e13"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.759375 1168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "543357c1819f4a98bbdf062ba53813f3"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "32ebd15a3d8a378311ebf9b6b358dd33"
server_key_iv: "c6fbe04d089ade73cec43158b67b7e13"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.762979 1168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:10.765373 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.766564 1168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:20:10.766680 1168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "543357c1819f4a98bbdf062ba53813f3"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "32ebd15a3d8a378311ebf9b6b358dd33"
server_key_iv: "c6fbe04d089ade73cec43158b67b7e13"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.766789 1168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:10.786398 1168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:10.787067 1168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:10.787264 1168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:10.787835 1168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:10.788751 1168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:10.788797 1168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.788866 1168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:10.788905 1168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.797660 1168 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:43355
I20260924 07:20:10.797688 1295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:43355 every 8 connection(s)
I20260924 07:20:10.798538 1168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:10.806280 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1168
I20260924 07:20:10.806423 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:10.806735 15870 external_mini_cluster.cc:1428] Setting key 18c1fb7017a01da93bc1d39c9972f719
I20260924 07:20:10.807847 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.800314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:53153 (local address 127.15.127.190:35859)
0924 07:20:10.800444 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:10.800447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:10.801520 (+ 1073us) server_negotiation.cc:407] Connection header received
0924 07:20:10.803035 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:10.803039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:10.803096 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:10.803214 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:10.804088 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.804619 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.805828 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.806052 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.806171 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.807345 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:10.807348 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:10.807351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:10.807364 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:10.807392 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:10.807401 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:10.807481 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:10.807643 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:20:10.807708 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:20:10.808794 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35859
--builtin_ntp_servers=127.15.127.148:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:20:10.808828 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:10.800657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35859 (local address 127.15.127.130:53153)
0924 07:20:10.801387 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:10.801429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:10.802777 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:10.803389 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:10.803396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:10.803480 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:10.803949 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.803962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.804757 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.804761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.805703 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:10.805713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:10.806212 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:10.806216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:10.806381 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:10.807082 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:10.807084 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:10.807124 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:10.807188 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:10.807483 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:10.807495 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:10.807502 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:20:10.807640 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 07:20:10.807925 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260924 07:20:10.809962 1296 heartbeater.cc:347] Connected to a master server at 127.15.127.190:35859
I20260924 07:20:10.810220 1296 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:10.810763 1296 heartbeater.cc:511] Master 127.15.127.190:35859 requested a full tablet report, sending...
I20260924 07:20:10.811796 977 ts_manager.cc:194] Registered new tserver with Master: 543357c1819f4a98bbdf062ba53813f3 (127.15.127.130:43355)
I20260924 07:20:10.812361 977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.130:53153
W20260924 07:20:10.917837 1299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:10.918066 1299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:10.918102 1299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:10.922070 1299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:10.922117 1299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:10.922186 1299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:20:10.926796 1299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:10.927850 1299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:10.929265 1299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:10.938894 1304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:10.938853 1305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:10.939054 1299 server_base.cc:1034] running on GCE node
W20260924 07:20:10.938869 1307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:10.939566 1299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:10.940316 1299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:10.941479 1299 hybrid_clock.cc:648] HybridClock initialized: now 1790234410941464 us; error 27 us; skew 500 ppm
I20260924 07:20:10.943325 1299 webserver.cc:460] Webserver started at http://127.15.127.131:35865/ using document root <none> and password file <none>
I20260924 07:20:10.943884 1299 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:10.943943 1299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:10.944176 1299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:10.946233 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "4867bc697c11422aaf3d110cc575ab6a"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "a151f079ee6dfb4b3e37fb7f9561174e"
server_key_iv: "9827fb429b7155953ab17ee2aaf38b59"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.946776 1299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "4867bc697c11422aaf3d110cc575ab6a"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "a151f079ee6dfb4b3e37fb7f9561174e"
server_key_iv: "9827fb429b7155953ab17ee2aaf38b59"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.950279 1299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:10.952706 1313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.954136 1299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:20:10.954267 1299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "4867bc697c11422aaf3d110cc575ab6a"
format_stamp: "Formatted at 2026-09-24 07:20:10 on dist-test-slave-zx46"
server_key: "a151f079ee6dfb4b3e37fb7f9561174e"
server_key_iv: "9827fb429b7155953ab17ee2aaf38b59"
server_key_version: "encryptionkey@0"
I20260924 07:20:10.954388 1299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:10.990654 1299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:10.991395 1299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:10.991592 1299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:10.992162 1299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:10.993242 1299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:10.993301 1299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:10.993358 1299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:10.993392 1299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:11.001802 1299 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:46371
I20260924 07:20:11.001833 1426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:46371 every 8 connection(s)
I20260924 07:20:11.002723 1299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:11.005578 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1299
I20260924 07:20:11.005699 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:11.006021 15870 external_mini_cluster.cc:1428] Setting key 8b7bda53c447d161141dd155bf4b3d64
I20260924 07:20:11.012173 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.004423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:46735 (local address 127.15.127.190:35859)
0924 07:20:11.004561 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.004565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.005723 (+ 1158us) server_negotiation.cc:407] Connection header received
0924 07:20:11.007466 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.007468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.007518 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.007585 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:11.008589 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.009166 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.010288 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.010514 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.010625 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.011687 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:11.011690 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:11.011693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:11.011713 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:11.011729 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:11.011743 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:11.011823 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.011983 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.012037 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:20:11.012953 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.004818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35859 (local address 127.15.127.131:46735)
0924 07:20:11.005543 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.005598 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:11.007275 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:11.007754 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:11.007761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:11.007844 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:11.008456 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.008469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.009298 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.009302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.010146 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.010155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.010661 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.010665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.010829 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.011437 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:11.011439 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:11.011476 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:11.011536 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:11.011825 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:11.011836 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:11.011844 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 07:20:11.011973 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 07:20:11.012229 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":146,"threads_started":1}
I20260924 07:20:11.013983 1427 heartbeater.cc:347] Connected to a master server at 127.15.127.190:35859
I20260924 07:20:11.014174 1427 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:11.014626 1427 heartbeater.cc:511] Master 127.15.127.190:35859 requested a full tablet report, sending...
I20260924 07:20:11.015537 977 ts_manager.cc:194] Registered new tserver with Master: 4867bc697c11422aaf3d110cc575ab6a (127.15.127.131:46371)
I20260924 07:20:11.016100 977 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.131:46735
I20260924 07:20:11.019547 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:11.024753 1016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.021433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48394 (local address 127.15.127.190:35859)
0924 07:20:11.021555 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.021560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.021690 (+ 130us) server_negotiation.cc:407] Connection header received
0924 07:20:11.021794 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.021796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.021831 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.021894 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:11.022134 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.022610 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.023386 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.023579 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.023648 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.024383 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:11.024386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:11.024387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:11.024397 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:11.024409 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:11.024415 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:11.024466 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.024556 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.024596 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:20:11.027385 977 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:11.030282 977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:11.043792 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.037724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:46371 (local address 127.0.0.1:55768)
0924 07:20:11.038532 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.038583 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:11.038712 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:11.039418 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:11.039424 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:11.039445 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:11.039602 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.039609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.041257 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.041262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.042433 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.042443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.043224 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.043228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.043337 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.043404 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 07:20:11.043501 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:20:11.043611 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":22,"spinlock_wait_cycles":3712,"thread_start_us":87,"threads_started":1}
I20260924 07:20:11.043790 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.038399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.15.127.131:46371)
0924 07:20:11.038941 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.038952 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.038976 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:20:11.039048 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.039053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.039271 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.039417 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:11.039712 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.040934 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.042553 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.043077 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.043219 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.043482 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.043530 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.043611 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":129,"threads_started":1}
I20260924 07:20:11.044678 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.038391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.15.127.130:43355)
0924 07:20:11.039235 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.039246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.039267 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:20:11.039337 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.039342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.039499 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.039647 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:11.039860 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.040954 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.042632 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.043409 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.043516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.043649 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.044401 (+ 752us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.044490 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":138,"threads_started":1}
I20260924 07:20:11.045081 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.038001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:43355 (local address 127.0.0.1:34188)
0924 07:20:11.038925 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.038938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:11.039002 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:11.039613 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:11.039616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:11.039633 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:11.039751 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.039756 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.041070 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.041073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.042502 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.042515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.043705 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.043709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.043834 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.043864 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 07:20:11.043924 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:20:11.043984 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":53,"thread_start_us":83,"threads_started":1}
I20260924 07:20:11.045928 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.037618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43931 (local address 127.0.0.1:51036)
0924 07:20:11.038532 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.038583 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:11.038712 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:11.039492 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:11.039495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:11.039514 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:11.041100 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.041115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.042598 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.042603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.043763 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.043771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.045542 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.045545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.045632 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.045653 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 07:20:11.045743 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 07:20:11.045804 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":1450,"spinlock_wait_cycles":6400,"thread_start_us":74,"threads_started":1}
I20260924 07:20:11.046083 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.037761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.15.127.129:43931)
0924 07:20:11.038149 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.038163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.038863 (+ 700us) server_negotiation.cc:407] Connection header received
0924 07:20:11.038954 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.038963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.039177 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.039344 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:11.041220 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.042470 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.044097 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.045111 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.045340 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.045488 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.045803 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.045902 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":100,"threads_started":1}
I20260924 07:20:11.047019 1361 tablet_service.cc:1467] Processing CreateTablet for tablet b44628fd4c314616b72c872db6728ff4 (DEFAULT_TABLE table=test-table [id=6fc8d6544c90461cb62d07367e8b23fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:11.048007 1361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44628fd4c314616b72c872db6728ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:11.048506 1099 tablet_service.cc:1467] Processing CreateTablet for tablet b44628fd4c314616b72c872db6728ff4 (DEFAULT_TABLE table=test-table [id=6fc8d6544c90461cb62d07367e8b23fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:11.049558 1099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44628fd4c314616b72c872db6728ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:11.054260 1444 tablet_bootstrap.cc:492] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Bootstrap starting.
I20260924 07:20:11.055271 1445 tablet_bootstrap.cc:492] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: Bootstrap starting.
I20260924 07:20:11.055022 1230 tablet_service.cc:1467] Processing CreateTablet for tablet b44628fd4c314616b72c872db6728ff4 (DEFAULT_TABLE table=test-table [id=6fc8d6544c90461cb62d07367e8b23fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:11.056015 1230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44628fd4c314616b72c872db6728ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:11.056552 1444 tablet_bootstrap.cc:654] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:11.057361 1445 tablet_bootstrap.cc:654] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:11.057448 1444 log.cc:824] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:11.058133 1445 log.cc:824] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:11.059654 1444 tablet_bootstrap.cc:492] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: No bootstrap required, opened a new log
I20260924 07:20:11.059865 1444 ts_tablet_manager.cc:1397] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:20:11.060478 1445 tablet_bootstrap.cc:492] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: No bootstrap required, opened a new log
I20260924 07:20:11.060662 1445 ts_tablet_manager.cc:1397] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:20:11.062971 1448 tablet_bootstrap.cc:492] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Bootstrap starting.
I20260924 07:20:11.063273 1444 raft_consensus.cc:348] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.063534 1444 raft_consensus.cc:374] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:11.063585 1444 raft_consensus.cc:729] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4867bc697c11422aaf3d110cc575ab6a, State: Initialized, Role: FOLLOWER
I20260924 07:20:11.064049 1444 consensus_queue.cc:260] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.064708 1427 heartbeater.cc:503] Master 127.15.127.190:35859 was elected leader, sending a full tablet report...
I20260924 07:20:11.064724 1445 raft_consensus.cc:348] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.064960 1445 raft_consensus.cc:374] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:11.065032 1445 raft_consensus.cc:729] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fb01800a0e4d01b55dafd1c40d844c, State: Initialized, Role: FOLLOWER
I20260924 07:20:11.065039 1444 ts_tablet_manager.cc:1428] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260924 07:20:11.065368 1448 tablet_bootstrap.cc:654] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:11.065470 1445 consensus_queue.cc:260] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.066160 1448 log.cc:824] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:11.067919 1448 tablet_bootstrap.cc:492] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: No bootstrap required, opened a new log
I20260924 07:20:11.067970 1165 heartbeater.cc:503] Master 127.15.127.190:35859 was elected leader, sending a full tablet report...
I20260924 07:20:11.068087 1448 ts_tablet_manager.cc:1397] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:20:11.068492 1445 ts_tablet_manager.cc:1428] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20260924 07:20:11.071877 1448 raft_consensus.cc:348] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.072093 1448 raft_consensus.cc:374] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:11.072157 1448 raft_consensus.cc:729] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543357c1819f4a98bbdf062ba53813f3, State: Initialized, Role: FOLLOWER
I20260924 07:20:11.072559 1448 consensus_queue.cc:260] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.073172 1296 heartbeater.cc:503] Master 127.15.127.190:35859 was elected leader, sending a full tablet report...
I20260924 07:20:11.073488 1448 ts_tablet_manager.cc:1428] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 07:20:11.108901 1166 tablet.cc:2369] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:11.113847 1452 raft_consensus.cc:482] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:11.114015 1452 raft_consensus.cc:504] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.114996 1452 leader_election.cc:288] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60fb01800a0e4d01b55dafd1c40d844c (127.15.127.129:43931), 4867bc697c11422aaf3d110cc575ab6a (127.15.127.131:46371)
I20260924 07:20:11.119803 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.115327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:35389 (local address 127.15.127.129:43931)
0924 07:20:11.115464 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.115469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.115728 (+ 259us) server_negotiation.cc:407] Connection header received
0924 07:20:11.115817 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.115820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.115874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.115941 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:11.116204 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.116938 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.118537 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.119127 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.119251 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.119300 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.119610 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.119677 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:20:11.119803 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.115335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:34153 (local address 127.15.127.131:46371)
0924 07:20:11.115464 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.115469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.115484 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:20:11.115621 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.115623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.115666 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.115770 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:11.116060 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.116856 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.118537 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.119127 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.119255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.119300 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.119610 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.119677 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:20:11.119868 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.115204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:46371 (local address 127.15.127.130:34153)
0924 07:20:11.115406 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.115422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:11.115507 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:11.115792 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:11.115795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:11.115812 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:11.115938 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.115947 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.116961 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.116963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.118411 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.118423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.119264 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.119268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.119440 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.119502 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 07:20:11.119581 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:20:11.119717 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":119}
I20260924 07:20:11.119884 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.115228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43931 (local address 127.15.127.130:35389)
0924 07:20:11.115635 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.115645 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:11.115719 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:11.115968 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:11.115974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:11.115994 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:11.116100 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.116105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.117092 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.117095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.118411 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:11.118423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.119264 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.119268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:11.119444 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.119504 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 07:20:11.119581 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:20:11.119713 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":140,"thread_start_us":135,"threads_started":1}
I20260924 07:20:11.120573 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44628fd4c314616b72c872db6728ff4" candidate_uuid: "543357c1819f4a98bbdf062ba53813f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60fb01800a0e4d01b55dafd1c40d844c" is_pre_election: true
I20260924 07:20:11.120579 1381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44628fd4c314616b72c872db6728ff4" candidate_uuid: "543357c1819f4a98bbdf062ba53813f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4867bc697c11422aaf3d110cc575ab6a" is_pre_election: true
I20260924 07:20:11.120894 1381 raft_consensus.cc:2454] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 543357c1819f4a98bbdf062ba53813f3 in term 0.
I20260924 07:20:11.120878 1119 raft_consensus.cc:2454] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 543357c1819f4a98bbdf062ba53813f3 in term 0.
I20260924 07:20:11.121455 1185 leader_election.cc:302] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4867bc697c11422aaf3d110cc575ab6a, 543357c1819f4a98bbdf062ba53813f3, 60fb01800a0e4d01b55dafd1c40d844c; no voters:
I20260924 07:20:11.121717 1452 raft_consensus.cc:2781] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:11.121805 1452 raft_consensus.cc:482] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:11.121837 1452 raft_consensus.cc:3037] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:11.122848 1452 raft_consensus.cc:504] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.123205 1452 leader_election.cc:288] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [CANDIDATE]: Term 1 election: Requested vote from peers 60fb01800a0e4d01b55dafd1c40d844c (127.15.127.129:43931), 4867bc697c11422aaf3d110cc575ab6a (127.15.127.131:46371)
I20260924 07:20:11.123481 1119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44628fd4c314616b72c872db6728ff4" candidate_uuid: "543357c1819f4a98bbdf062ba53813f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60fb01800a0e4d01b55dafd1c40d844c"
I20260924 07:20:11.123528 1381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44628fd4c314616b72c872db6728ff4" candidate_uuid: "543357c1819f4a98bbdf062ba53813f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4867bc697c11422aaf3d110cc575ab6a"
I20260924 07:20:11.123608 1119 raft_consensus.cc:3037] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:11.123669 1381 raft_consensus.cc:3037] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:11.124607 1381 raft_consensus.cc:2454] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543357c1819f4a98bbdf062ba53813f3 in term 1.
I20260924 07:20:11.124611 1119 raft_consensus.cc:2454] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 543357c1819f4a98bbdf062ba53813f3 in term 1.
I20260924 07:20:11.124920 1186 leader_election.cc:302] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4867bc697c11422aaf3d110cc575ab6a, 543357c1819f4a98bbdf062ba53813f3; no voters:
I20260924 07:20:11.125101 1452 raft_consensus.cc:2781] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:11.125366 1452 raft_consensus.cc:686] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 1 LEADER]: Becoming Leader. State: Replica: 543357c1819f4a98bbdf062ba53813f3, State: Running, Role: LEADER
I20260924 07:20:11.125695 1452 consensus_queue.cc:237] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } }
I20260924 07:20:11.128643 977 catalog_manager.cc:5495] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 543357c1819f4a98bbdf062ba53813f3 (127.15.127.130). New cstate: current_term: 1 leader_uuid: "543357c1819f4a98bbdf062ba53813f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "543357c1819f4a98bbdf062ba53813f3" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 43355 } health_report { overall_health: HEALTHY } } }
I20260924 07:20:11.138655 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.135791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.15.127.130:43355)
0924 07:20:11.135909 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:11.135912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:11.135922 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:20:11.135975 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:11.135977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:11.136022 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:11.136100 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:11.136289 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.136710 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.137536 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:11.137736 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:11.137811 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:11.138035 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:11.138393 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:11.138477 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:11.138526 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:20:11.138567 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:20:11.143946 1381 raft_consensus.cc:1261] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 1 FOLLOWER]: Refusing update from remote peer 543357c1819f4a98bbdf062ba53813f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:11.143980 1119 raft_consensus.cc:1261] T b44628fd4c314616b72c872db6728ff4 P 60fb01800a0e4d01b55dafd1c40d844c [term 1 FOLLOWER]: Refusing update from remote peer 543357c1819f4a98bbdf062ba53813f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:11.144618 1452 consensus_queue.cc:1035] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4867bc697c11422aaf3d110cc575ab6a" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:11.144800 1454 consensus_queue.cc:1035] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60fb01800a0e4d01b55dafd1c40d844c" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 43931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:11.154081 1458 mvcc.cc:205] Tried to move back new op lower bound from 7332800148038479872 to 7332800147973214208. Current Snapshot: MvccSnapshot[applied={T|T < 7332800148038479872}]
I20260924 07:20:11.160215 976 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:11.160356 976 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:11.162559 976 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e029b82a9ba244d4a875aca2d85d7b8e: Sending DeleteTablet for 3 replicas of tablet b44628fd4c314616b72c872db6728ff4
I20260924 07:20:11.163316 1099 tablet_service.cc:1514] Processing DeleteTablet for tablet b44628fd4c314616b72c872db6728ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:11 UTC) from {username='slave'} at 127.0.0.1:51036
I20260924 07:20:11.163304 1361 tablet_service.cc:1514] Processing DeleteTablet for tablet b44628fd4c314616b72c872db6728ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:11 UTC) from {username='slave'} at 127.0.0.1:55768
I20260924 07:20:11.163362 1230 tablet_service.cc:1514] Processing DeleteTablet for tablet b44628fd4c314616b72c872db6728ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:11 UTC) from {username='slave'} at 127.0.0.1:34188
I20260924 07:20:11.163800 1468 tablet_replica.cc:330] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: stopping tablet replica
I20260924 07:20:11.164152 1468 raft_consensus.cc:2229] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:11.164494 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1037
I20260924 07:20:11.164516 1468 raft_consensus.cc:2258] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:11.165086 1467 tablet_replica.cc:330] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: stopping tablet replica
I20260924 07:20:11.165385 1467 raft_consensus.cc:2229] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:11.165657 1467 raft_consensus.cc:2258] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:11.166851 1467 ts_tablet_manager.cc:1905] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:11.166879 1468 ts_tablet_manager.cc:1905] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:11.169390 1467 ts_tablet_manager.cc:1918] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:11.169574 1467 log.cc:1196] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/b44628fd4c314616b72c872db6728ff4
I20260924 07:20:11.169957 1467 ts_tablet_manager.cc:1939] T b44628fd4c314616b72c872db6728ff4 P 4867bc697c11422aaf3d110cc575ab6a: Deleting consensus metadata
I20260924 07:20:11.170068 1468 ts_tablet_manager.cc:1918] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:11.170172 1468 log.cc:1196] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/b44628fd4c314616b72c872db6728ff4
I20260924 07:20:11.170485 1468 ts_tablet_manager.cc:1939] T b44628fd4c314616b72c872db6728ff4 P 543357c1819f4a98bbdf062ba53813f3: Deleting consensus metadata
I20260924 07:20:11.171584 962 catalog_manager.cc:4841] TS 543357c1819f4a98bbdf062ba53813f3 (127.15.127.130:43355): tablet b44628fd4c314616b72c872db6728ff4 (table test-table [id=6fc8d6544c90461cb62d07367e8b23fb]) successfully deleted
I20260924 07:20:11.171584 964 catalog_manager.cc:4841] TS 4867bc697c11422aaf3d110cc575ab6a (127.15.127.131:46371): tablet b44628fd4c314616b72c872db6728ff4 (table test-table [id=6fc8d6544c90461cb62d07367e8b23fb]) successfully deleted
W20260924 07:20:11.173389 963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:43931 (error 108)
I20260924 07:20:11.174233 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1168
I20260924 07:20:11.174465 1441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:11.174128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:43931 (local address 127.0.0.1:51038)
0924 07:20:11.174286 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:11.174362 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:43931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 07:20:11.176081 963 catalog_manager.cc:4584] TS 60fb01800a0e4d01b55dafd1c40d844c (127.15.127.129:43931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b44628fd4c314616b72c872db6728ff4: Network error: Client connection negotiation failed: client connection to 127.15.127.129:43931: connect: Connection refused (error 111)
I20260924 07:20:11.180447 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1299
I20260924 07:20:11.185981 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 946
2026-09-24T07:20:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1070 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:11 dist-test-slave-zx46 krb5kdc[1473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:11 dist-test-slave-zx46 krb5kdc[1473](info): set up 2 sockets
Sep 24 07:20:11 dist-test-slave-zx46 krb5kdc[1473](info): commencing operation
krb5kdc: starting...
W20260924 07:20:12.228835 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:12 dist-test-slave-zx46 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:12.348683 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:35601
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:35601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:12.457221 1486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:12.457526 1486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:12.457602 1486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:12.461598 1486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:12.461683 1486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:12.461723 1486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:12.461777 1486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:12.461817 1486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:12.466974 1486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:35601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:35601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:12.467989 1486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:12.468981 1486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:12.477322 1491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:12.477389 1494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:12.477451 1486 server_base.cc:1034] running on GCE node
W20260924 07:20:12.477576 1492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:12.478158 1486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:12.479111 1486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:12.480311 1486 hybrid_clock.cc:648] HybridClock initialized: now 1790234412480290 us; error 33 us; skew 500 ppm
Sep 24 07:20:12 dist-test-slave-zx46 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234412, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:12.488848 1486 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:12.489809 1486 webserver.cc:460] Webserver started at http://127.15.127.190:43997/ using document root <none> and password file <none>
I20260924 07:20:12.490355 1486 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:12.490448 1486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:12.490679 1486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:12.492682 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "5287b15a1e5a4b68b278c80676cb09f1"
format_stamp: "Formatted at 2026-09-24 07:20:12 on dist-test-slave-zx46"
server_key: "0903d70deec1f71545fddbedfa974735"
server_key_iv: "efc9eefa1171fe989e397d914c23fc92"
server_key_version: "encryptionkey@0"
I20260924 07:20:12.493285 1486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "5287b15a1e5a4b68b278c80676cb09f1"
format_stamp: "Formatted at 2026-09-24 07:20:12 on dist-test-slave-zx46"
server_key: "0903d70deec1f71545fddbedfa974735"
server_key_iv: "efc9eefa1171fe989e397d914c23fc92"
server_key_version: "encryptionkey@0"
I20260924 07:20:12.496793 1486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 07:20:12.499084 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:12.500231 1486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:20:12.500379 1486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "5287b15a1e5a4b68b278c80676cb09f1"
format_stamp: "Formatted at 2026-09-24 07:20:12 on dist-test-slave-zx46"
server_key: "0903d70deec1f71545fddbedfa974735"
server_key_iv: "efc9eefa1171fe989e397d914c23fc92"
server_key_version: "encryptionkey@0"
I20260924 07:20:12.500494 1486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:12.517707 1486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:12.520901 1486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:12.521130 1486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:12.528225 1486 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:35601
I20260924 07:20:12.528263 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:35601 every 8 connection(s)
I20260924 07:20:12.529160 1486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:12.531782 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:12.534683 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1486
I20260924 07:20:12.534842 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:12.535163 15870 external_mini_cluster.cc:1428] Setting key 2329fd27c4ebdd3f6fd7f1c7d0bd6d1f
I20260924 07:20:12.537860 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1: Bootstrap starting.
I20260924 07:20:12.539911 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:12.540653 1554 log.cc:824] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:12.542461 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1: No bootstrap required, opened a new log
Sep 24 07:20:12 dist-test-slave-zx46 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:12.546303 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5287b15a1e5a4b68b278c80676cb09f1" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35601 } }
I20260924 07:20:12.546622 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:12.546689 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5287b15a1e5a4b68b278c80676cb09f1, State: Initialized, Role: FOLLOWER
I20260924 07:20:12.547212 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5287b15a1e5a4b68b278c80676cb09f1" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35601 } }
I20260924 07:20:12.547379 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:12.547451 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:12.547546 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:12.548435 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5287b15a1e5a4b68b278c80676cb09f1" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35601 } }
I20260924 07:20:12.548784 1554 leader_election.cc:302] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5287b15a1e5a4b68b278c80676cb09f1; no voters:
I20260924 07:20:12.549172 1554 leader_election.cc:288] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:12.549274 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:12.549477 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [term 1 LEADER]: Becoming Leader. State: Replica: 5287b15a1e5a4b68b278c80676cb09f1, State: Running, Role: LEADER
I20260924 07:20:12.549593 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:12.536566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.15.127.190:35601)
0924 07:20:12.536995 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:12.537062 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:12.537118 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:20:12.537734 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:12.537749 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:12.538114 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:12.538430 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:12.539227 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:12.540217 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:12.541181 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:12.541649 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:12.541819 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:12.544550 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:12.544555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:12.544568 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:12.544594 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:12.546408 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:12.546934 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:12.546943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:12.546947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:12.547048 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:12.547369 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:12.547378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:12.547379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:12.547819 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:12.548026 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:12.548253 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:20:12.548521 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":140,"threads_started":1}
I20260924 07:20:12.549857 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5287b15a1e5a4b68b278c80676cb09f1" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35601 } }
I20260924 07:20:12.550407 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:12.551373 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5287b15a1e5a4b68b278c80676cb09f1. Latest consensus state: current_term: 1 leader_uuid: "5287b15a1e5a4b68b278c80676cb09f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5287b15a1e5a4b68b278c80676cb09f1" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35601 } } }
I20260924 07:20:12.551779 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:12.551687 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5287b15a1e5a4b68b278c80676cb09f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5287b15a1e5a4b68b278c80676cb09f1" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 35601 } } }
I20260924 07:20:12.552173 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:12.552423 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:12.555521 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:12.560635 1569 catalog_manager.cc:1347] Generated new cluster ID: e51ec3d21df54ce385e54f59c4ec85cb
I20260924 07:20:12.560694 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:12.589299 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:12.590268 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:12.597820 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1: Generated new TSK 0
I20260924 07:20:12.598507 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:20:12.632532 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35601
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:12.741685 1580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:12.741956 1580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:12.742004 1580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:12.746438 1580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:12.746497 1580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:12.746595 1580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:20:12.752203 1580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:12.753365 1580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:12.754410 1580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:12.764292 1585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:12.764364 1588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:12.764515 1580 server_base.cc:1034] running on GCE node
W20260924 07:20:12.764379 1586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:12.765110 1580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:12.765751 1580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:12.766937 1580 hybrid_clock.cc:648] HybridClock initialized: now 1790234412766920 us; error 37 us; skew 500 ppm
Sep 24 07:20:12 dist-test-slave-zx46 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234412, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:12.769865 1580 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:20:12.770783 1580 webserver.cc:460] Webserver started at http://127.15.127.129:34389/ using document root <none> and password file <none>
I20260924 07:20:12.771347 1580 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:12.771405 1580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:12.771633 1580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:12.773742 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "26e3f37770e843cf836f4f940cda2add"
format_stamp: "Formatted at 2026-09-24 07:20:12 on dist-test-slave-zx46"
server_key: "01c17617fe5a83d3972bd37f65950024"
server_key_iv: "b2ba3601a556b973b90c80e7d5849873"
server_key_version: "encryptionkey@0"
I20260924 07:20:12.774307 1580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "26e3f37770e843cf836f4f940cda2add"
format_stamp: "Formatted at 2026-09-24 07:20:12 on dist-test-slave-zx46"
server_key: "01c17617fe5a83d3972bd37f65950024"
server_key_iv: "b2ba3601a556b973b90c80e7d5849873"
server_key_version: "encryptionkey@0"
I20260924 07:20:12.777909 1580 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:12.780380 1595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:12.781755 1580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:20:12.781893 1580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "26e3f37770e843cf836f4f940cda2add"
format_stamp: "Formatted at 2026-09-24 07:20:12 on dist-test-slave-zx46"
server_key: "01c17617fe5a83d3972bd37f65950024"
server_key_iv: "b2ba3601a556b973b90c80e7d5849873"
server_key_version: "encryptionkey@0"
I20260924 07:20:12.782009 1580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:12.832266 1580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:12.836079 1580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:12.836320 1580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:12.836972 1580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:12.837898 1580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:12.837949 1580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:12.838025 1580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:12.838060 1580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:12.847913 1580 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:44565
I20260924 07:20:12.847936 1708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:44565 every 8 connection(s)
I20260924 07:20:12.848858 1580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:12.849192 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1580
I20260924 07:20:12.849337 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:12.849704 15870 external_mini_cluster.cc:1428] Setting key 2beb5c3dd470a9f9bd01f9554fbf2a0e
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
Sep 24 07:20:12 dist-test-slave-zx46 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234412, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
I20260924 07:20:12.864845 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:12.850930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:52125 (local address 127.15.127.190:35601)
0924 07:20:12.851076 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:12.851082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:12.852151 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 07:20:12.854120 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:12.854126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:12.854186 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:12.854285 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:12.855521 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:12.856569 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:12.857852 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:12.858218 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:12.858340 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:12.860966 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:12.860970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:12.860972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:12.860996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:12.862433 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:12.863048 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:12.863051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:12.863052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:12.863126 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:12.863516 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:12.863518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:12.863519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:12.863719 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:12.863796 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:12.864480 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 07:20:12.864628 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:20:12.865754 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:12.851426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35601 (local address 127.15.127.129:52125)
0924 07:20:12.851976 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:12.852036 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:12.853897 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:12.854451 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:12.854458 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:12.854863 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:12.855360 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:12.855373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:12.856730 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:12.856734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:12.857705 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:12.857715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:12.858368 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:12.858371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:12.858511 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:12.859131 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:12.859147 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:12.860626 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:12.862579 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:12.862587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:12.862603 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:12.862943 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:12.863224 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:12.863229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:12.863233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:12.863424 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:12.863826 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:12.863835 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:12.864186 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 07:20:12.864397 (+ 211us) client_negotiation.cc:240] Negotiation successful
0924 07:20:12.864652 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":154,"threads_started":1}
I20260924 07:20:12.867849 1709 heartbeater.cc:347] Connected to a master server at 127.15.127.190:35601
I20260924 07:20:12.868117 1709 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:12.868613 1709 heartbeater.cc:511] Master 127.15.127.190:35601 requested a full tablet report, sending...
I20260924 07:20:12.870175 1515 ts_manager.cc:194] Registered new tserver with Master: 26e3f37770e843cf836f4f940cda2add (127.15.127.129:44565)
I20260924 07:20:12.871272 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:52125
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:20:12.880033 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35601
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:12.988133 1714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:12.988365 1714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:12.988406 1714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:12.992328 1714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:12.992376 1714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:12.992432 1714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:20:12.997500 1714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:12.998441 1714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:12.999433 1714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:13.008639 1720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.008639 1719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:13.008708 1714 server_base.cc:1034] running on GCE node
W20260924 07:20:13.008643 1722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:13.009382 1714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:13.009969 1714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:13.011170 1714 hybrid_clock.cc:648] HybridClock initialized: now 1790234413011147 us; error 29 us; skew 500 ppm
Sep 24 07:20:13 dist-test-slave-zx46 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234413, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:13.013918 1714 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:20:13.014788 1714 webserver.cc:460] Webserver started at http://127.15.127.130:38399/ using document root <none> and password file <none>
I20260924 07:20:13.015378 1714 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:13.015439 1714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:13.015672 1714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:13.017758 1714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "c98df0f965454bf8b181a8ca812de498"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "74bfb9d3b4dccaef3e9ad0dfddb6ebf7"
server_key_iv: "3a0a80ac7e800519f9579516a062f227"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.018337 1714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "c98df0f965454bf8b181a8ca812de498"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "74bfb9d3b4dccaef3e9ad0dfddb6ebf7"
server_key_iv: "3a0a80ac7e800519f9579516a062f227"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.021947 1714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:13.024586 1729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.025816 1714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:20:13.025939 1714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "c98df0f965454bf8b181a8ca812de498"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "74bfb9d3b4dccaef3e9ad0dfddb6ebf7"
server_key_iv: "3a0a80ac7e800519f9579516a062f227"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.026041 1714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:13.049281 1714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:13.052500 1714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:13.052687 1714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:13.053354 1714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:13.054111 1714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:13.054155 1714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.054193 1714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:13.054214 1714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.062685 1714 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:39561
I20260924 07:20:13.062711 1842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:39561 every 8 connection(s)
I20260924 07:20:13.063550 1714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:13.066747 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1714
I20260924 07:20:13.066903 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:13.067256 15870 external_mini_cluster.cc:1428] Setting key 5e9593f99ef6e0c514b0faf5f79cc1dd
Sep 24 07:20:13 dist-test-slave-zx46 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234413, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
I20260924 07:20:13.077839 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.065389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:46441 (local address 127.15.127.190:35601)
0924 07:20:13.065526 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.065530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.066381 (+ 851us) server_negotiation.cc:407] Connection header received
0924 07:20:13.068061 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.068064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.068117 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.068196 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:13.069674 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.070228 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.071378 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.071612 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.071719 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.074210 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:13.074213 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:13.074215 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:13.074242 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:13.075758 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:13.076286 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:13.076290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:13.076292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:13.076372 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:13.076716 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:13.076721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:13.076723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:13.076900 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:13.077009 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.077553 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.077673 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:20:13.078792 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.065786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35601 (local address 127.15.127.130:46441)
0924 07:20:13.066240 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.066282 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.067854 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.068363 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.068369 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.068790 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:13.069517 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.069539 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.070355 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.070359 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.071244 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.071253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.071778 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.071782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.071944 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.072538 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:13.072553 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:13.074035 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:13.075890 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:13.075895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:13.075909 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:13.076197 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:13.076473 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:13.076476 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:13.076478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:13.076615 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:13.077053 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:13.077061 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:13.077312 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.077498 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.077822 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
I20260924 07:20:13.079967 1843 heartbeater.cc:347] Connected to a master server at 127.15.127.190:35601
I20260924 07:20:13.080248 1843 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:13.080850 1843 heartbeater.cc:511] Master 127.15.127.190:35601 requested a full tablet report, sending...
I20260924 07:20:13.082054 1515 ts_manager.cc:194] Registered new tserver with Master: c98df0f965454bf8b181a8ca812de498 (127.15.127.130:39561)
I20260924 07:20:13.082603 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:46441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:20:13.094231 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35601
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:33965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:13.201169 1848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:13.201448 1848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:13.201519 1848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:13.205574 1848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:13.205649 1848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:13.205737 1848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:20:13.210834 1848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:33965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:35601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:13.211807 1848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:13.212796 1848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:13.221897 1853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.221897 1854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.221899 1856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:13.222122 1848 server_base.cc:1034] running on GCE node
I20260924 07:20:13.222556 1848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:13.223169 1848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:13.224360 1848 hybrid_clock.cc:648] HybridClock initialized: now 1790234413224345 us; error 27 us; skew 500 ppm
Sep 24 07:20:13 dist-test-slave-zx46 krb5kdc[1473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234413, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:13.227105 1848 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:20:13.228009 1848 webserver.cc:460] Webserver started at http://127.15.127.131:39369/ using document root <none> and password file <none>
I20260924 07:20:13.228741 1848 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:13.228837 1848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:13.229125 1848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:13.231194 1848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "7c1a91cc893c427ea3929c78f5b39a6e"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "028100a704fcd6f4be10c5e48e4eebf6"
server_key_iv: "96778b55e763aa9bbdfb8290ee0286b3"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.231684 1848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "7c1a91cc893c427ea3929c78f5b39a6e"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "028100a704fcd6f4be10c5e48e4eebf6"
server_key_iv: "96778b55e763aa9bbdfb8290ee0286b3"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.235145 1848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:13.237372 1863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.238550 1848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:13.238652 1848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "7c1a91cc893c427ea3929c78f5b39a6e"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "028100a704fcd6f4be10c5e48e4eebf6"
server_key_iv: "96778b55e763aa9bbdfb8290ee0286b3"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.238731 1848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:13.263027 1848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:13.265915 1848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:13.266065 1848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:13.266594 1848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:13.267364 1848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:13.267409 1848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.267447 1848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:13.267462 1848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.276142 1848 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:36003
I20260924 07:20:13.276175 1976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:36003 every 8 connection(s)
I20260924 07:20:13.276974 1848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:13.279999 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 1848
I20260924 07:20:13.280153 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:13.280525 15870 external_mini_cluster.cc:1428] Setting key 28ab2a8d2ed6fcde943aefcea464c1dc
Sep 24 07:20:13 dist-test-slave-zx46 krb5kdc[1473](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234413, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:13.291100 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.278700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:55293 (local address 127.15.127.190:35601)
0924 07:20:13.278861 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.278866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.279742 (+ 876us) server_negotiation.cc:407] Connection header received
0924 07:20:13.281446 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.281449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.281504 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.281604 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:13.283123 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.283636 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.284756 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.284971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.285118 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.287644 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:13.287647 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:13.287650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:13.287684 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:13.289147 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:13.289647 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:13.289651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:13.289653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:13.289739 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:13.290068 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:13.290073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:13.290075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:13.290253 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:13.290359 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.290826 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.290936 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:20:13.291920 1979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.279178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:35601 (local address 127.15.127.131:55293)
0924 07:20:13.279582 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.279624 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.281236 (+ 1612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.281953 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.281960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.282409 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:13.282984 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.282997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.283776 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.283779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.284630 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.284639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.285161 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.285165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.285334 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.285911 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:13.285925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:13.287479 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:13.289278 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:13.289283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:13.289297 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:13.289542 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:13.289831 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:13.289836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:13.289838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:13.289973 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:13.290356 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:13.290362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:13.290600 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.290786 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.291079 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260924 07:20:13.293149 1977 heartbeater.cc:347] Connected to a master server at 127.15.127.190:35601
I20260924 07:20:13.293392 1977 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:13.293951 1977 heartbeater.cc:511] Master 127.15.127.190:35601 requested a full tablet report, sending...
I20260924 07:20:13.295054 1515 ts_manager.cc:194] Registered new tserver with Master: 7c1a91cc893c427ea3929c78f5b39a6e (127.15.127.131:36003)
I20260924 07:20:13.295588 1515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:55293
I20260924 07:20:13.306685 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:13.317056 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.308643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.15.127.190:35601)
0924 07:20:13.308849 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.308855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.309090 (+ 235us) server_negotiation.cc:407] Connection header received
0924 07:20:13.309261 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.309265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.309325 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.309437 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:13.310187 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.310716 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.311607 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.311833 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.311949 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.313580 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:13.313585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:13.313588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:13.313623 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:13.314928 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:13.315669 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:13.315673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:13.315675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:13.315755 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:13.316123 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:13.316126 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:13.316128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:13.316335 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:13.316493 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.316754 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.316846 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260924 07:20:13.319907 1515 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:13.323019 1515 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:13.336895 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.331028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49106 (local address 127.15.127.129:44565)
0924 07:20:13.331887 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.331907 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.331933 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:20:13.332012 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.332018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.332238 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.332548 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:13.332815 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.334015 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.335600 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.336256 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.336391 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.336523 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.336664 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.336764 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":95,"threads_started":1}
I20260924 07:20:13.336889 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.330820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:44565 (local address 127.0.0.1:49106)
0924 07:20:13.331229 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.331285 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.331415 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.332421 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.332427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.332464 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:13.332677 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.332687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.334149 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.334152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.335498 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.335507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.336379 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.336382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.336488 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.336523 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.336636 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.336744 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":116,"spinlock_wait_cycles":7296,"thread_start_us":111,"threads_started":1}
I20260924 07:20:13.337477 1991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.331535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:39561 (local address 127.0.0.1:37170)
0924 07:20:13.332395 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.332407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.332770 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.333300 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.333303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.333321 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:13.333400 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.333406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.334585 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.334589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.335727 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.335737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.336888 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.336891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.337141 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.337194 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.337265 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.337358 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":128,"thread_start_us":86,"threads_started":1}
I20260924 07:20:13.337579 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.331614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.15.127.131:36003)
0924 07:20:13.332016 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.332026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.332044 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:20:13.332105 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.332109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.332245 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.332539 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:13.333018 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.333901 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.335761 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.336591 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.336709 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.336825 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.337339 (+ 514us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.337440 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":120,"threads_started":1}
I20260924 07:20:13.337864 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.331176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:36003 (local address 127.0.0.1:43998)
0924 07:20:13.331437 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.331452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.331800 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.332421 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.332427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.332463 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:13.332895 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.332900 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.334026 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.334031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.335651 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.335665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.336888 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.336891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.337152 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.337207 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.337265 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.337748 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":598,"spinlock_wait_cycles":4864,"thread_start_us":114,"threads_started":1}
I20260924 07:20:13.337872 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.331815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.15.127.130:39561)
0924 07:20:13.332139 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.332153 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.332699 (+ 546us) server_negotiation.cc:407] Connection header received
0924 07:20:13.332923 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.332929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.333154 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.333306 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:13.333510 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.334448 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.335831 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.336591 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.336709 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.336825 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.337651 (+ 826us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.337751 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":147,"threads_started":1}
I20260924 07:20:13.340493 1643 tablet_service.cc:1467] Processing CreateTablet for tablet 5f730b8ecd16460f999ecf5c9a9c53ed (DEFAULT_TABLE table=test-table [id=9d28933fb7db4d1596f5f734c35ec51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:13.340492 1911 tablet_service.cc:1467] Processing CreateTablet for tablet 5f730b8ecd16460f999ecf5c9a9c53ed (DEFAULT_TABLE table=test-table [id=9d28933fb7db4d1596f5f734c35ec51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:13.341449 1643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f730b8ecd16460f999ecf5c9a9c53ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:13.341488 1911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f730b8ecd16460f999ecf5c9a9c53ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:13.342841 1777 tablet_service.cc:1467] Processing CreateTablet for tablet 5f730b8ecd16460f999ecf5c9a9c53ed (DEFAULT_TABLE table=test-table [id=9d28933fb7db4d1596f5f734c35ec51e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:13.343752 1777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f730b8ecd16460f999ecf5c9a9c53ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:13.347335 1994 tablet_bootstrap.cc:492] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Bootstrap starting.
I20260924 07:20:13.349167 1995 tablet_bootstrap.cc:492] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Bootstrap starting.
I20260924 07:20:13.349610 1994 tablet_bootstrap.cc:654] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:13.350543 1996 tablet_bootstrap.cc:492] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: Bootstrap starting.
I20260924 07:20:13.351226 1995 tablet_bootstrap.cc:654] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:13.351197 1994 log.cc:824] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:13.352594 1995 log.cc:824] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:13.352741 1996 tablet_bootstrap.cc:654] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:13.353987 1996 log.cc:824] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:13.354301 1994 tablet_bootstrap.cc:492] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: No bootstrap required, opened a new log
I20260924 07:20:13.354490 1994 ts_tablet_manager.cc:1397] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 07:20:13.355903 1996 tablet_bootstrap.cc:492] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: No bootstrap required, opened a new log
I20260924 07:20:13.356079 1996 ts_tablet_manager.cc:1397] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 07:20:13.358542 1995 tablet_bootstrap.cc:492] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: No bootstrap required, opened a new log
I20260924 07:20:13.358413 1994 raft_consensus.cc:348] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.358686 1994 raft_consensus.cc:374] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:13.358743 1994 raft_consensus.cc:729] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e3f37770e843cf836f4f940cda2add, State: Initialized, Role: FOLLOWER
I20260924 07:20:13.358744 1995 ts_tablet_manager.cc:1397] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260924 07:20:13.359189 1994 consensus_queue.cc:260] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.359848 1709 heartbeater.cc:503] Master 127.15.127.190:35601 was elected leader, sending a full tablet report...
I20260924 07:20:13.359786 1996 raft_consensus.cc:348] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.360005 1996 raft_consensus.cc:374] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:13.360051 1996 raft_consensus.cc:729] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98df0f965454bf8b181a8ca812de498, State: Initialized, Role: FOLLOWER
I20260924 07:20:13.360200 1994 ts_tablet_manager.cc:1428] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260924 07:20:13.360481 1996 consensus_queue.cc:260] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.361231 1843 heartbeater.cc:503] Master 127.15.127.190:35601 was elected leader, sending a full tablet report...
I20260924 07:20:13.361500 1996 ts_tablet_manager.cc:1428] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 07:20:13.362623 1995 raft_consensus.cc:348] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.362895 1995 raft_consensus.cc:374] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:13.362962 1995 raft_consensus.cc:729] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1a91cc893c427ea3929c78f5b39a6e, State: Initialized, Role: FOLLOWER
I20260924 07:20:13.363355 1995 consensus_queue.cc:260] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.364008 1977 heartbeater.cc:503] Master 127.15.127.190:35601 was elected leader, sending a full tablet report...
I20260924 07:20:13.364315 1995 ts_tablet_manager.cc:1428] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:20:13.458938 2001 raft_consensus.cc:482] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:13.459138 2001 raft_consensus.cc:504] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.460280 2001 leader_election.cc:288] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26e3f37770e843cf836f4f940cda2add (127.15.127.129:44565), 7c1a91cc893c427ea3929c78f5b39a6e (127.15.127.131:36003)
I20260924 07:20:13.465447 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.460639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:51107 (local address 127.15.127.129:44565)
0924 07:20:13.460760 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.460766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.460784 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:20:13.460976 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.460978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.461084 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.461163 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:13.461520 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.462323 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.464115 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.464729 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.464844 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.464895 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.465243 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.465320 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:20:13.465447 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.460619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:46201 (local address 127.15.127.131:36003)
0924 07:20:13.460747 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.460752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.461034 (+ 282us) server_negotiation.cc:407] Connection header received
0924 07:20:13.461220 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.461224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.461278 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.461353 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:13.461661 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.462384 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.464117 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.464719 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.464843 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.464894 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.465247 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.465317 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:20:13.465484 1845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.460502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:44565 (local address 127.15.127.130:51107)
0924 07:20:13.460720 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.460736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.460844 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.461237 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.461240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.461253 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:13.461397 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.461404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.462465 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.462469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.463985 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.463996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.464848 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.464851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.465062 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.465135 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.465220 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.465319 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":220}
I20260924 07:20:13.465492 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.460519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:36003 (local address 127.15.127.130:46201)
0924 07:20:13.460928 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.460939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.461099 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.461372 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.461375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.461395 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:13.461532 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.461539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.462485 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.462487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.463993 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.463999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.464848 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.464851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.465080 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.465132 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.465212 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.465319 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":222,"thread_start_us":124,"threads_started":1}
I20260924 07:20:13.466181 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f730b8ecd16460f999ecf5c9a9c53ed" candidate_uuid: "c98df0f965454bf8b181a8ca812de498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" is_pre_election: true
I20260924 07:20:13.466190 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f730b8ecd16460f999ecf5c9a9c53ed" candidate_uuid: "c98df0f965454bf8b181a8ca812de498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e3f37770e843cf836f4f940cda2add" is_pre_election: true
I20260924 07:20:13.466485 1663 raft_consensus.cc:2454] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c98df0f965454bf8b181a8ca812de498 in term 0.
I20260924 07:20:13.466490 1931 raft_consensus.cc:2454] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c98df0f965454bf8b181a8ca812de498 in term 0.
I20260924 07:20:13.466992 1731 leader_election.cc:302] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26e3f37770e843cf836f4f940cda2add, 7c1a91cc893c427ea3929c78f5b39a6e, c98df0f965454bf8b181a8ca812de498; no voters:
I20260924 07:20:13.467310 2001 raft_consensus.cc:2781] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:13.467406 2001 raft_consensus.cc:482] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:13.467468 2001 raft_consensus.cc:3037] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:13.468511 2001 raft_consensus.cc:504] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.468859 2001 leader_election.cc:288] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [CANDIDATE]: Term 1 election: Requested vote from peers 26e3f37770e843cf836f4f940cda2add (127.15.127.129:44565), 7c1a91cc893c427ea3929c78f5b39a6e (127.15.127.131:36003)
I20260924 07:20:13.469228 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f730b8ecd16460f999ecf5c9a9c53ed" candidate_uuid: "c98df0f965454bf8b181a8ca812de498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1a91cc893c427ea3929c78f5b39a6e"
I20260924 07:20:13.469235 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f730b8ecd16460f999ecf5c9a9c53ed" candidate_uuid: "c98df0f965454bf8b181a8ca812de498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e3f37770e843cf836f4f940cda2add"
I20260924 07:20:13.469357 1663 raft_consensus.cc:3037] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:13.469324 1931 raft_consensus.cc:3037] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:13.470371 1663 raft_consensus.cc:2454] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c98df0f965454bf8b181a8ca812de498 in term 1.
I20260924 07:20:13.470377 1931 raft_consensus.cc:2454] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c98df0f965454bf8b181a8ca812de498 in term 1.
I20260924 07:20:13.470680 1732 leader_election.cc:302] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c1a91cc893c427ea3929c78f5b39a6e, c98df0f965454bf8b181a8ca812de498; no voters:
I20260924 07:20:13.470894 2001 raft_consensus.cc:2781] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:13.471148 2001 raft_consensus.cc:686] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 1 LEADER]: Becoming Leader. State: Replica: c98df0f965454bf8b181a8ca812de498, State: Running, Role: LEADER
I20260924 07:20:13.471477 2001 consensus_queue.cc:237] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } }
I20260924 07:20:13.474773 1515 catalog_manager.cc:5495] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 reported cstate change: term changed from 0 to 1, leader changed from <none> to c98df0f965454bf8b181a8ca812de498 (127.15.127.130). New cstate: current_term: 1 leader_uuid: "c98df0f965454bf8b181a8ca812de498" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c98df0f965454bf8b181a8ca812de498" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 39561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 } health_report { overall_health: UNKNOWN } } }
I20260924 07:20:13.500025 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.496481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.15.127.130:39561)
0924 07:20:13.496612 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.496616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.496628 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:13.496694 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.496696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.496761 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.496857 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:13.497129 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.497637 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.498507 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.498744 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.498857 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.499164 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:13.499567 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:13.499734 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.499836 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.499875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:20:13.506074 1663 raft_consensus.cc:1261] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 1 FOLLOWER]: Refusing update from remote peer c98df0f965454bf8b181a8ca812de498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:13.506596 1931 raft_consensus.cc:1261] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 1 FOLLOWER]: Refusing update from remote peer c98df0f965454bf8b181a8ca812de498: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:13.506841 2004 consensus_queue.cc:1035] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e3f37770e843cf836f4f940cda2add" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 44565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:13.507295 2001 consensus_queue.cc:1035] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c1a91cc893c427ea3929c78f5b39a6e" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 36003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:13.514621 2007 mvcc.cc:205] Tried to move back new op lower bound from 7332800157712891904 to 7332800157581713408. Current Snapshot: MvccSnapshot[applied={T|T < 7332800157712891904}]
I20260924 07:20:13.516757 2009 mvcc.cc:205] Tried to move back new op lower bound from 7332800157712891904 to 7332800157581713408. Current Snapshot: MvccSnapshot[applied={T|T < 7332800157712891904}]
I20260924 07:20:13.517750 2008 mvcc.cc:205] Tried to move back new op lower bound from 7332800157712891904 to 7332800157581713408. Current Snapshot: MvccSnapshot[applied={T|T < 7332800157712891904}]
I20260924 07:20:13.523931 1515 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:13.524102 1515 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:13.526429 1515 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5287b15a1e5a4b68b278c80676cb09f1: Sending DeleteTablet for 3 replicas of tablet 5f730b8ecd16460f999ecf5c9a9c53ed
I20260924 07:20:13.527184 1911 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f730b8ecd16460f999ecf5c9a9c53ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:13 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:43998
I20260924 07:20:13.527181 1777 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f730b8ecd16460f999ecf5c9a9c53ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:13 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:37170
I20260924 07:20:13.527298 1643 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f730b8ecd16460f999ecf5c9a9c53ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:13 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:49106
I20260924 07:20:13.527769 2019 tablet_replica.cc:330] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: stopping tablet replica
I20260924 07:20:13.528182 2019 raft_consensus.cc:2229] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:13.528538 2019 raft_consensus.cc:2258] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:13.529896 2019 ts_tablet_manager.cc:1905] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:13.532115 2019 ts_tablet_manager.cc:1918] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:13.532227 2019 log.cc:1196] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-0/wal/wals/5f730b8ecd16460f999ecf5c9a9c53ed
I20260924 07:20:13.532507 2019 ts_tablet_manager.cc:1939] T 5f730b8ecd16460f999ecf5c9a9c53ed P 26e3f37770e843cf836f4f940cda2add: Deleting consensus metadata
I20260924 07:20:13.533504 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1580
I20260924 07:20:13.534122 1503 catalog_manager.cc:4841] TS 26e3f37770e843cf836f4f940cda2add (127.15.127.129:44565): tablet 5f730b8ecd16460f999ecf5c9a9c53ed (table test-table [id=9d28933fb7db4d1596f5f734c35ec51e]) successfully deleted
I20260924 07:20:13.537227 2018 tablet_replica.cc:330] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: stopping tablet replica
I20260924 07:20:13.537552 2018 raft_consensus.cc:2229] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:13.537864 2018 raft_consensus.cc:2258] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:13.539415 2018 ts_tablet_manager.cc:1905] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:13.542167 2017 tablet_replica.cc:330] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498: stopping tablet replica
I20260924 07:20:13.542484 2017 raft_consensus.cc:2229] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:13.542863 2017 raft_consensus.cc:2258] T 5f730b8ecd16460f999ecf5c9a9c53ed P c98df0f965454bf8b181a8ca812de498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:13.543778 2018 ts_tablet_manager.cc:1918] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:13.543872 2018 log.cc:1196] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/5f730b8ecd16460f999ecf5c9a9c53ed
I20260924 07:20:13.543875 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1714
I20260924 07:20:13.545248 2018 ts_tablet_manager.cc:1939] T 5f730b8ecd16460f999ecf5c9a9c53ed P 7c1a91cc893c427ea3929c78f5b39a6e: Deleting consensus metadata
I20260924 07:20:13.546327 1504 catalog_manager.cc:4841] TS 7c1a91cc893c427ea3929c78f5b39a6e (127.15.127.131:36003): tablet 5f730b8ecd16460f999ecf5c9a9c53ed (table test-table [id=9d28933fb7db4d1596f5f734c35ec51e]) successfully deleted
W20260924 07:20:13.548993 1503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.130:39561 (error 108)
I20260924 07:20:13.549223 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1848
I20260924 07:20:13.550537 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.550156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:39561 (local address 127.0.0.1:37192)
0924 07:20:13.550365 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.550439 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:39561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260924 07:20:13.550696 1503 catalog_manager.cc:4584] TS c98df0f965454bf8b181a8ca812de498 (127.15.127.130:39561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f730b8ecd16460f999ecf5c9a9c53ed: Network error: Client connection negotiation failed: client connection to 127.15.127.130:39561: connect: Connection refused (error 111)
I20260924 07:20:13.554911 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 1486
2026-09-24T07:20:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2369 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T07:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:13Z Disabled control of system clock
I20260924 07:20:13.584734 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41665
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:41665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:20:13.691072 2024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:13.691313 2024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:13.691346 2024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:13.695200 2024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:13.695247 2024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:13.695262 2024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:13.695279 2024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:13.695293 2024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:13.699796 2024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:41665
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:13.700719 2024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:13.701994 2024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:13.710440 2029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.710440 2032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.710500 2030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:13.710848 2024 server_base.cc:1034] running on GCE node
I20260924 07:20:13.711320 2024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:13.712378 2024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:13.713543 2024 hybrid_clock.cc:648] HybridClock initialized: now 1790234413713505 us; error 46 us; skew 500 ppm
I20260924 07:20:13.715536 2024 webserver.cc:460] Webserver started at http://127.15.127.190:45777/ using document root <none> and password file <none>
I20260924 07:20:13.716113 2024 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:13.716176 2024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:13.716408 2024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:13.718464 2024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "c8ad0e0b024d47be85194464075df500"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "1af0163b3be55307d56516ad6a87b327"
server_key_iv: "88bab45b9568706310496de44b3f87ee"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.719019 2024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "c8ad0e0b024d47be85194464075df500"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "1af0163b3be55307d56516ad6a87b327"
server_key_iv: "88bab45b9568706310496de44b3f87ee"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.722651 2024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:13.725214 2038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.726432 2024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:13.726608 2024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "c8ad0e0b024d47be85194464075df500"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "1af0163b3be55307d56516ad6a87b327"
server_key_iv: "88bab45b9568706310496de44b3f87ee"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.726717 2024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:13.743141 2024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:13.743885 2024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:13.744119 2024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:13.751716 2024 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:41665
I20260924 07:20:13.751718 2090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:41665 every 8 connection(s)
I20260924 07:20:13.752734 2024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:13.755919 2091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:13.760742 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2024
I20260924 07:20:13.760879 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:13.761215 15870 external_mini_cluster.cc:1428] Setting key 30da3c1111cf792dff4f3c8740ad990d
I20260924 07:20:13.762694 2091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500: Bootstrap starting.
I20260924 07:20:13.764901 2091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:13.765671 2091 log.cc:824] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:13.767445 2091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500: No bootstrap required, opened a new log
I20260924 07:20:13.770494 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.762648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.15.127.190:41665)
0924 07:20:13.763294 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.763309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.763343 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:13.763989 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.764006 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.764395 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.764740 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:13.765228 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.766267 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.767150 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.767672 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.767860 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.768743 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:13.768748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:13.768760 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:13.769245 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:13.769299 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:13.769319 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:13.769429 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.769543 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.769801 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":124,"threads_started":1}
I20260924 07:20:13.771049 2091 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ad0e0b024d47be85194464075df500" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41665 } }
I20260924 07:20:13.771284 2091 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:13.771346 2091 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ad0e0b024d47be85194464075df500, State: Initialized, Role: FOLLOWER
I20260924 07:20:13.771772 2091 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ad0e0b024d47be85194464075df500" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41665 } }
I20260924 07:20:13.771921 2091 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:13.772008 2091 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:13.772150 2091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:13.773164 2091 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ad0e0b024d47be85194464075df500" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41665 } }
I20260924 07:20:13.773481 2091 leader_election.cc:302] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ad0e0b024d47be85194464075df500; no voters:
I20260924 07:20:13.773766 2091 leader_election.cc:288] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:13.773895 2096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:13.774113 2096 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [term 1 LEADER]: Becoming Leader. State: Replica: c8ad0e0b024d47be85194464075df500, State: Running, Role: LEADER
I20260924 07:20:13.774421 2096 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ad0e0b024d47be85194464075df500" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41665 } }
I20260924 07:20:13.775004 2091 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:13.776124 2097 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8ad0e0b024d47be85194464075df500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ad0e0b024d47be85194464075df500" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41665 } } }
I20260924 07:20:13.776106 2098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ad0e0b024d47be85194464075df500. Latest consensus state: current_term: 1 leader_uuid: "c8ad0e0b024d47be85194464075df500" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ad0e0b024d47be85194464075df500" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41665 } } }
I20260924 07:20:13.776250 2098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:13.776249 2097 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:13.776669 2105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:13.779570 2105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:13.784578 2105 catalog_manager.cc:1347] Generated new cluster ID: d8d7c7a3c0774f85b7c0731c5e24d067
I20260924 07:20:13.784639 2105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:13.797204 2105 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:13.798098 2105 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:13.804373 2105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500: Generated new TSK 0
I20260924 07:20:13.805075 2105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:13.808543 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41665
--builtin_ntp_servers=127.15.127.148:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:20:13.918924 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:13.919169 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:13.919206 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:13.923085 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:13.923135 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:13.923188 2115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:20:13.927711 2115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:13.928663 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:13.930063 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:13.939664 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.939692 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:13.939694 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:13.939992 2115 server_base.cc:1034] running on GCE node
I20260924 07:20:13.940428 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:13.941150 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:13.942340 2115 hybrid_clock.cc:648] HybridClock initialized: now 1790234413942320 us; error 32 us; skew 500 ppm
I20260924 07:20:13.944289 2115 webserver.cc:460] Webserver started at http://127.15.127.129:43775/ using document root <none> and password file <none>
I20260924 07:20:13.944864 2115 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:13.944967 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:13.945220 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:13.947228 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "a1d6299401c84db1a5ff5729a626f34e"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "7c2a00a413906cb2a35ec10989868dfd"
server_key_iv: "4b4507188de1bb28e03170217dbe474c"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.947784 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "a1d6299401c84db1a5ff5729a626f34e"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "7c2a00a413906cb2a35ec10989868dfd"
server_key_iv: "4b4507188de1bb28e03170217dbe474c"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.951431 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 07:20:13.953987 2129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.955235 2115 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:13.955369 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "a1d6299401c84db1a5ff5729a626f34e"
format_stamp: "Formatted at 2026-09-24 07:20:13 on dist-test-slave-zx46"
server_key: "7c2a00a413906cb2a35ec10989868dfd"
server_key_iv: "4b4507188de1bb28e03170217dbe474c"
server_key_version: "encryptionkey@0"
I20260924 07:20:13.955482 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:13.970738 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:13.971503 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:13.971706 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:13.972303 2115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:13.973325 2115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:13.973394 2115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.973465 2115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:13.973500 2115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:13.982771 2115 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:45323
I20260924 07:20:13.982837 2242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:45323 every 8 connection(s)
I20260924 07:20:13.983727 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:13.985091 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2115
I20260924 07:20:13.985235 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:13.985555 15870 external_mini_cluster.cc:1428] Setting key 56002a8e39ba46988974eb23a3aca7d7
I20260924 07:20:13.987604 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41665
--builtin_ntp_servers=127.15.127.148:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:20:13.993304 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.985650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:39973 (local address 127.15.127.190:41665)
0924 07:20:13.985788 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:13.985793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:13.986993 (+ 1200us) server_negotiation.cc:407] Connection header received
0924 07:20:13.988499 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:13.988503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:13.988558 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:13.988637 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:13.989575 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.990132 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.991286 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.991533 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.991647 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.992751 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:13.992755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:13.992758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:13.992796 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:13.992819 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:13.992827 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:13.992901 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:13.993109 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:20:13.993168 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:20:13.994177 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:13.986121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41665 (local address 127.15.127.129:39973)
0924 07:20:13.986862 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:13.986912 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:13.988285 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:13.988806 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:13.988813 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:13.988899 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:13.989413 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.989427 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.990270 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.990275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.991146 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:13.991155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:13.991683 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:13.991687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:13.991853 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:13.992486 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:13.992488 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:13.992527 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:13.992586 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:13.992915 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:13.992926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:13.992933 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:20:13.993102 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 07:20:13.993385 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":156,"threads_started":1}
I20260924 07:20:13.996218 2243 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41665
I20260924 07:20:13.996510 2243 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:13.997226 2243 heartbeater.cc:511] Master 127.15.127.190:41665 requested a full tablet report, sending...
I20260924 07:20:13.998903 2055 ts_manager.cc:194] Registered new tserver with Master: a1d6299401c84db1a5ff5729a626f34e (127.15.127.129:45323)
I20260924 07:20:14.000373 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.129:39973
W20260924 07:20:14.100636 2246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:14.100916 2246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:14.100955 2246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:14.105158 2246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:14.105209 2246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:14.105264 2246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:20:14.110015 2246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:14.111025 2246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:14.112324 2246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:14.121711 2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:14.121716 2254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:14.121754 2252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:14.122157 2246 server_base.cc:1034] running on GCE node
I20260924 07:20:14.122566 2246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:14.123281 2246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:14.124476 2246 hybrid_clock.cc:648] HybridClock initialized: now 1790234414124456 us; error 42 us; skew 500 ppm
I20260924 07:20:14.126459 2246 webserver.cc:460] Webserver started at http://127.15.127.130:35877/ using document root <none> and password file <none>
I20260924 07:20:14.127066 2246 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:14.127153 2246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:14.127391 2246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:14.129504 2246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "36de5315f22c452fab25e261209feeee"
format_stamp: "Formatted at 2026-09-24 07:20:14 on dist-test-slave-zx46"
server_key: "6f562b64285c73e480311b9bf0dbd980"
server_key_iv: "6480c3ec327185dc33ceadc48db5fba8"
server_key_version: "encryptionkey@0"
I20260924 07:20:14.130067 2246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "36de5315f22c452fab25e261209feeee"
format_stamp: "Formatted at 2026-09-24 07:20:14 on dist-test-slave-zx46"
server_key: "6f562b64285c73e480311b9bf0dbd980"
server_key_iv: "6480c3ec327185dc33ceadc48db5fba8"
server_key_version: "encryptionkey@0"
I20260924 07:20:14.133749 2246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:14.136169 2260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:14.137475 2246 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:14.137588 2246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "36de5315f22c452fab25e261209feeee"
format_stamp: "Formatted at 2026-09-24 07:20:14 on dist-test-slave-zx46"
server_key: "6f562b64285c73e480311b9bf0dbd980"
server_key_iv: "6480c3ec327185dc33ceadc48db5fba8"
server_key_version: "encryptionkey@0"
I20260924 07:20:14.137681 2246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:14.166646 2246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:14.167372 2246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:14.167587 2246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:14.168192 2246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:14.169113 2246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:14.169158 2246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:14.169221 2246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:14.169269 2246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:14.178629 2246 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:34239
I20260924 07:20:14.178681 2373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:34239 every 8 connection(s)
I20260924 07:20:14.179523 2246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:14.183874 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2246
I20260924 07:20:14.184008 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:14.184360 15870 external_mini_cluster.cc:1428] Setting key 457c014e027659ceaa1b31b1daf1f3aa
I20260924 07:20:14.186349 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41665
--builtin_ntp_servers=127.15.127.148:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:20:14.189055 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.181305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:43499 (local address 127.15.127.190:41665)
0924 07:20:14.181456 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.181461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.182525 (+ 1064us) server_negotiation.cc:407] Connection header received
0924 07:20:14.183914 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.183916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.183993 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.184094 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:14.185272 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.185889 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.187108 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.187333 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.187447 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.188546 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:14.188549 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:14.188552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:14.188572 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:14.188589 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:14.188597 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:14.188677 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.188838 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.188896 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:20:14.189875 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.181746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41665 (local address 127.15.127.130:43499)
0924 07:20:14.182379 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.182417 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.183690 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.184301 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.184310 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.184416 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:14.185103 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.185139 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.186019 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.186022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.186979 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.186989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.187484 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.187487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.187646 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.188290 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:14.188292 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:14.188332 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:14.188392 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:14.188678 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:14.188690 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:14.188699 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.188836 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.189133 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":121,"threads_started":1}
I20260924 07:20:14.190941 2374 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41665
I20260924 07:20:14.191184 2374 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:14.191671 2374 heartbeater.cc:511] Master 127.15.127.190:41665 requested a full tablet report, sending...
I20260924 07:20:14.192699 2055 ts_manager.cc:194] Registered new tserver with Master: 36de5315f22c452fab25e261209feeee (127.15.127.130:34239)
I20260924 07:20:14.193593 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.130:43499
W20260924 07:20:14.295331 2377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:14.295578 2377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:14.295637 2377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:14.299289 2377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:14.299329 2377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:14.299378 2377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:20:14.303885 2377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:14.304944 2377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:14.306278 2377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:14.315753 2383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:14.315763 2382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:14.315797 2377 server_base.cc:1034] running on GCE node
W20260924 07:20:14.315758 2385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:14.316370 2377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:14.317052 2377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:14.318248 2377 hybrid_clock.cc:648] HybridClock initialized: now 1790234414318217 us; error 43 us; skew 500 ppm
I20260924 07:20:14.319980 2377 webserver.cc:460] Webserver started at http://127.15.127.131:37393/ using document root <none> and password file <none>
I20260924 07:20:14.320576 2377 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:14.320683 2377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:14.320909 2377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:14.322930 2377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "5709442c75e74dd39217f2d0a08562b6"
format_stamp: "Formatted at 2026-09-24 07:20:14 on dist-test-slave-zx46"
server_key: "f2f4372ac212d7483c95969b0648b952"
server_key_iv: "cb94fb27529552bb4acd1076ec741810"
server_key_version: "encryptionkey@0"
I20260924 07:20:14.323484 2377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "5709442c75e74dd39217f2d0a08562b6"
format_stamp: "Formatted at 2026-09-24 07:20:14 on dist-test-slave-zx46"
server_key: "f2f4372ac212d7483c95969b0648b952"
server_key_iv: "cb94fb27529552bb4acd1076ec741810"
server_key_version: "encryptionkey@0"
I20260924 07:20:14.327054 2377 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:14.329337 2391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:14.330508 2377 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:14.330638 2377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "5709442c75e74dd39217f2d0a08562b6"
format_stamp: "Formatted at 2026-09-24 07:20:14 on dist-test-slave-zx46"
server_key: "f2f4372ac212d7483c95969b0648b952"
server_key_iv: "cb94fb27529552bb4acd1076ec741810"
server_key_version: "encryptionkey@0"
I20260924 07:20:14.330746 2377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:14.346437 2377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:14.347067 2377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:14.347255 2377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:14.347828 2377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:14.348752 2377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:14.348826 2377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:14.348891 2377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:14.348940 2377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:14.358042 2377 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:33607
I20260924 07:20:14.358076 2504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:33607 every 8 connection(s)
I20260924 07:20:14.358961 2377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:14.362396 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2377
I20260924 07:20:14.362524 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:14.362812 15870 external_mini_cluster.cc:1428] Setting key d8de1d00e838fd6216bfbcb12c629378
I20260924 07:20:14.368428 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.360886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:55317 (local address 127.15.127.190:41665)
0924 07:20:14.361065 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.361070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.362165 (+ 1095us) server_negotiation.cc:407] Connection header received
0924 07:20:14.363638 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.363642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.363694 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.363802 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:14.364893 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.365516 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.366672 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.366892 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.366975 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.367989 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:14.367992 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:14.367994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:14.368005 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:14.368024 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:14.368032 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:14.368123 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.368259 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.368313 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:20:14.369274 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.361362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41665 (local address 127.15.127.131:55317)
0924 07:20:14.361992 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.362029 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.363394 (+ 1365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.363940 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.363947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.364068 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:14.364731 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.364752 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.365634 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.365637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.366542 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.366551 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.367016 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.367019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.367153 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.367700 (+ 547us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:14.367702 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:14.367745 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:14.367816 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:14.368137 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:14.368143 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:14.368147 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.368238 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.368441 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260924 07:20:14.370337 2505 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41665
I20260924 07:20:14.370565 2505 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:14.371035 2505 heartbeater.cc:511] Master 127.15.127.190:41665 requested a full tablet report, sending...
I20260924 07:20:14.371974 2055 ts_manager.cc:194] Registered new tserver with Master: 5709442c75e74dd39217f2d0a08562b6 (127.15.127.131:33607)
I20260924 07:20:14.372548 2055 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.131:55317
I20260924 07:20:14.376675 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:14.382056 2094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.378621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.15.127.190:41665)
0924 07:20:14.378752 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.378755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.378916 (+ 161us) server_negotiation.cc:407] Connection header received
0924 07:20:14.379002 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.379004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.379036 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.379117 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:14.379361 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.379841 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.380609 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.380825 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.380897 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.381696 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:14.381700 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:14.381701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:14.381713 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:14.381730 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:14.381738 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:14.381831 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.381925 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.381967 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:20:14.384645 2055 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46058:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:14.387544 2055 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:14.401928 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.395684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:34239 (local address 127.0.0.1:50800)
0924 07:20:14.396780 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.396793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.396871 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.397960 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.397962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.397980 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:14.398053 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.398058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.399079 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.399082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.400302 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.400312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.401379 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.401383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.401485 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.401524 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.401651 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.401774 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":68,"thread_start_us":89,"threads_started":1}
I20260924 07:20:14.401965 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.396395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.15.127.130:34239)
0924 07:20:14.397481 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.397495 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.397518 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:20:14.397630 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.397635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.397815 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.397971 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:14.398157 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.398959 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.400421 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.401262 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.401380 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.401547 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.401677 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.401782 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":976,"thread_start_us":116,"threads_started":1}
I20260924 07:20:14.402348 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.395603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:33607 (local address 127.0.0.1:56046)
0924 07:20:14.396242 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.396290 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.396418 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.397258 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.397266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.397292 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:14.397416 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.397423 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.398880 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.398883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.400837 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.400845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.401992 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.401996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.402081 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.402101 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.402177 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.402240 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":884,"thread_start_us":64,"threads_started":1}
I20260924 07:20:14.402685 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.396711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40708 (local address 127.15.127.129:45323)
0924 07:20:14.397078 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.397087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.397159 (+ 72us) server_negotiation.cc:407] Connection header received
0924 07:20:14.397331 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.397335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.397490 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.397649 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:14.397836 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.398961 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.400485 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.401258 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.401380 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.401547 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.401999 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.402108 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":156,"threads_started":1}
I20260924 07:20:14.402746 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.395734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.15.127.131:33607)
0924 07:20:14.396030 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.396043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.396568 (+ 525us) server_negotiation.cc:407] Connection header received
0924 07:20:14.396646 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.396651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.396866 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.397035 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:14.397540 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.398765 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.400950 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.401667 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.401769 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.401930 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.402515 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.402609 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":124,"threads_started":1}
I20260924 07:20:14.403633 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.396317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45323 (local address 127.0.0.1:40708)
0924 07:20:14.397086 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.397099 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.397191 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.397637 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.397640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.397657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:14.397733 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.397738 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.399212 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.399216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.400386 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.400397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.401617 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.401621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.401720 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.401742 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.401814 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.401888 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":7,"thread_start_us":84,"threads_started":1}
I20260924 07:20:14.405227 2439 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1ce5bf777740fd95eeb25cac48e825 (DEFAULT_TABLE table=test-table [id=1fe19f71046d4639abeabd6aa835bdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:14.405190 2308 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1ce5bf777740fd95eeb25cac48e825 (DEFAULT_TABLE table=test-table [id=1fe19f71046d4639abeabd6aa835bdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:14.406203 2308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ce5bf777740fd95eeb25cac48e825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:14.406222 2439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ce5bf777740fd95eeb25cac48e825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:14.406176 2177 tablet_service.cc:1467] Processing CreateTablet for tablet 1c1ce5bf777740fd95eeb25cac48e825 (DEFAULT_TABLE table=test-table [id=1fe19f71046d4639abeabd6aa835bdb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:14.407085 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c1ce5bf777740fd95eeb25cac48e825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:14.412906 2522 tablet_bootstrap.cc:492] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Bootstrap starting.
I20260924 07:20:14.414179 2524 tablet_bootstrap.cc:492] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: Bootstrap starting.
I20260924 07:20:14.414467 2523 tablet_bootstrap.cc:492] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Bootstrap starting.
I20260924 07:20:14.415573 2522 tablet_bootstrap.cc:654] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:14.415927 2524 tablet_bootstrap.cc:654] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:14.416467 2523 tablet_bootstrap.cc:654] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:14.416466 2522 log.cc:824] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:14.416628 2524 log.cc:824] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:14.417295 2523 log.cc:824] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:14.418529 2522 tablet_bootstrap.cc:492] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: No bootstrap required, opened a new log
I20260924 07:20:14.418676 2524 tablet_bootstrap.cc:492] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: No bootstrap required, opened a new log
I20260924 07:20:14.418726 2522 ts_tablet_manager.cc:1397] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:20:14.418870 2524 ts_tablet_manager.cc:1397] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:20:14.419201 2523 tablet_bootstrap.cc:492] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: No bootstrap required, opened a new log
I20260924 07:20:14.419377 2523 ts_tablet_manager.cc:1397] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 07:20:14.423480 2523 raft_consensus.cc:348] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.423509 2522 raft_consensus.cc:348] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.423756 2523 raft_consensus.cc:374] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:14.423807 2522 raft_consensus.cc:374] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:14.423813 2523 raft_consensus.cc:729] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36de5315f22c452fab25e261209feeee, State: Initialized, Role: FOLLOWER
I20260924 07:20:14.423856 2522 raft_consensus.cc:729] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5709442c75e74dd39217f2d0a08562b6, State: Initialized, Role: FOLLOWER
I20260924 07:20:14.423224 2524 raft_consensus.cc:348] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.424315 2523 consensus_queue.cc:260] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.424429 2524 raft_consensus.cc:374] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:14.424362 2522 consensus_queue.cc:260] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.424495 2524 raft_consensus.cc:729] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1d6299401c84db1a5ff5729a626f34e, State: Initialized, Role: FOLLOWER
I20260924 07:20:14.424923 2524 consensus_queue.cc:260] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.425372 2523 ts_tablet_manager.cc:1428] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 07:20:14.425505 2374 heartbeater.cc:503] Master 127.15.127.190:41665 was elected leader, sending a full tablet report...
I20260924 07:20:14.425910 2243 heartbeater.cc:503] Master 127.15.127.190:41665 was elected leader, sending a full tablet report...
I20260924 07:20:14.425920 2524 ts_tablet_manager.cc:1428] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:20:14.426322 2522 ts_tablet_manager.cc:1428] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 07:20:14.426661 2505 heartbeater.cc:503] Master 127.15.127.190:41665 was elected leader, sending a full tablet report...
W20260924 07:20:14.430379 2375 tablet.cc:2369] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:20:14.484911 2244 tablet.cc:2369] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:14.530742 2528 raft_consensus.cc:482] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:14.530972 2528 raft_consensus.cc:504] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.532148 2528 leader_election.cc:288] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5709442c75e74dd39217f2d0a08562b6 (127.15.127.131:33607), a1d6299401c84db1a5ff5729a626f34e (127.15.127.129:45323)
I20260924 07:20:14.536876 2376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.532408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:33607 (local address 127.15.127.130:58905)
0924 07:20:14.532539 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.532556 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.532654 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.533042 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.533046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.533067 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:14.533203 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.533213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.534286 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.534291 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.535605 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.535627 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.536466 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.536472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.536577 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.536631 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.536690 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.536745 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":44}
I20260924 07:20:14.536952 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.532510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:58905 (local address 127.15.127.131:33607)
0924 07:20:14.532636 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.532654 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.532675 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:20:14.532748 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.532754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.532822 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.532914 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:14.533317 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.534146 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.535728 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.536342 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.536464 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.536510 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.536738 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.536822 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:20:14.537075 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.532702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45323 (local address 127.15.127.130:48691)
0924 07:20:14.533042 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.533054 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:14.533147 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:14.533403 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:14.533405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:14.533419 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:14.533506 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.533509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.534452 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.534455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.535594 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:14.535606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.536592 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.536595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:14.536740 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.536764 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:20:14.536839 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:20:14.536910 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":30,"thread_start_us":153,"threads_started":1}
I20260924 07:20:14.537132 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.532843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:48691 (local address 127.15.127.129:45323)
0924 07:20:14.532968 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.532972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.533129 (+ 157us) server_negotiation.cc:407] Connection header received
0924 07:20:14.533237 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.533239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.533307 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.533387 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:14.533609 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.534366 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.535724 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.536365 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.536464 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.536510 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.536874 (+ 364us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.536953 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:20:14.537564 2459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ce5bf777740fd95eeb25cac48e825" candidate_uuid: "36de5315f22c452fab25e261209feeee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5709442c75e74dd39217f2d0a08562b6" is_pre_election: true
I20260924 07:20:14.537927 2459 raft_consensus.cc:2454] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36de5315f22c452fab25e261209feeee in term 0.
I20260924 07:20:14.538036 2197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ce5bf777740fd95eeb25cac48e825" candidate_uuid: "36de5315f22c452fab25e261209feeee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d6299401c84db1a5ff5729a626f34e" is_pre_election: true
I20260924 07:20:14.538309 2197 raft_consensus.cc:2454] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36de5315f22c452fab25e261209feeee in term 0.
I20260924 07:20:14.538481 2262 leader_election.cc:302] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36de5315f22c452fab25e261209feeee, 5709442c75e74dd39217f2d0a08562b6; no voters:
I20260924 07:20:14.538807 2528 raft_consensus.cc:2781] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:14.538898 2528 raft_consensus.cc:482] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:14.538954 2528 raft_consensus.cc:3037] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:14.539948 2528 raft_consensus.cc:504] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.540277 2528 leader_election.cc:288] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [CANDIDATE]: Term 1 election: Requested vote from peers 5709442c75e74dd39217f2d0a08562b6 (127.15.127.131:33607), a1d6299401c84db1a5ff5729a626f34e (127.15.127.129:45323)
I20260924 07:20:14.540597 2459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ce5bf777740fd95eeb25cac48e825" candidate_uuid: "36de5315f22c452fab25e261209feeee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5709442c75e74dd39217f2d0a08562b6"
I20260924 07:20:14.540643 2197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c1ce5bf777740fd95eeb25cac48e825" candidate_uuid: "36de5315f22c452fab25e261209feeee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1d6299401c84db1a5ff5729a626f34e"
I20260924 07:20:14.540771 2459 raft_consensus.cc:3037] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:14.540783 2197 raft_consensus.cc:3037] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:14.541961 2459 raft_consensus.cc:2454] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36de5315f22c452fab25e261209feeee in term 1.
I20260924 07:20:14.542042 2197 raft_consensus.cc:2454] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36de5315f22c452fab25e261209feeee in term 1.
I20260924 07:20:14.542290 2262 leader_election.cc:302] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36de5315f22c452fab25e261209feeee, 5709442c75e74dd39217f2d0a08562b6; no voters:
I20260924 07:20:14.542447 2528 raft_consensus.cc:2781] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:14.542706 2528 raft_consensus.cc:686] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 1 LEADER]: Becoming Leader. State: Replica: 36de5315f22c452fab25e261209feeee, State: Running, Role: LEADER
I20260924 07:20:14.543038 2528 consensus_queue.cc:237] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } }
I20260924 07:20:14.546476 2054 catalog_manager.cc:5495] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee reported cstate change: term changed from 0 to 1, leader changed from <none> to 36de5315f22c452fab25e261209feeee (127.15.127.130). New cstate: current_term: 1 leader_uuid: "36de5315f22c452fab25e261209feeee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36de5315f22c452fab25e261209feeee" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 34239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 } health_report { overall_health: UNKNOWN } } }
I20260924 07:20:14.590332 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.586416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.15.127.130:34239)
0924 07:20:14.586589 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:14.586596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:14.586614 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:20:14.586697 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:14.586699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:14.586759 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:14.586884 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:14.587202 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.587731 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.588674 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:14.588893 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:14.589037 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:14.589381 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:14.589811 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:14.590001 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:14.590127 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:20:14.590167 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:20:14.596328 2197 raft_consensus.cc:1261] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 1 FOLLOWER]: Refusing update from remote peer 36de5315f22c452fab25e261209feeee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:14.596361 2459 raft_consensus.cc:1261] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 1 FOLLOWER]: Refusing update from remote peer 36de5315f22c452fab25e261209feeee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:14.597072 2528 consensus_queue.cc:1035] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1d6299401c84db1a5ff5729a626f34e" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 45323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:14.597266 2532 consensus_queue.cc:1035] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [LEADER]: Connected to new peer: Peer: permanent_uuid: "5709442c75e74dd39217f2d0a08562b6" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 33607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:14.605461 2536 mvcc.cc:205] Tried to move back new op lower bound from 7332800162178920448 to 7332800161970810880. Current Snapshot: MvccSnapshot[applied={T|T < 7332800162178920448}]
I20260924 07:20:14.613916 2053 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:14.614078 2053 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46058:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:14.616361 2053 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8ad0e0b024d47be85194464075df500: Sending DeleteTablet for 3 replicas of tablet 1c1ce5bf777740fd95eeb25cac48e825
I20260924 07:20:14.617197 2308 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c1ce5bf777740fd95eeb25cac48e825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:14 UTC) from {username='slave'} at 127.0.0.1:50800
I20260924 07:20:14.617259 2439 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c1ce5bf777740fd95eeb25cac48e825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:14 UTC) from {username='slave'} at 127.0.0.1:56046
I20260924 07:20:14.617496 2177 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c1ce5bf777740fd95eeb25cac48e825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:14 UTC) from {username='slave'} at 127.0.0.1:40708
I20260924 07:20:14.617794 2546 tablet_replica.cc:330] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: stopping tablet replica
I20260924 07:20:14.617976 2547 tablet_replica.cc:330] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e: stopping tablet replica
I20260924 07:20:14.618227 2546 raft_consensus.cc:2229] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:14.618284 2547 raft_consensus.cc:2229] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:14.618577 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2115
I20260924 07:20:14.618635 2547 raft_consensus.cc:2258] T 1c1ce5bf777740fd95eeb25cac48e825 P a1d6299401c84db1a5ff5729a626f34e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:14.618752 2546 raft_consensus.cc:2258] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:14.620532 2546 ts_tablet_manager.cc:1905] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:14.622890 2545 tablet_replica.cc:330] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: stopping tablet replica
I20260924 07:20:14.623198 2545 raft_consensus.cc:2229] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:14.623342 2546 ts_tablet_manager.cc:1918] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:14.623430 2546 log.cc:1196] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/1c1ce5bf777740fd95eeb25cac48e825
I20260924 07:20:14.623560 2545 raft_consensus.cc:2258] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:14.623734 2546 ts_tablet_manager.cc:1939] T 1c1ce5bf777740fd95eeb25cac48e825 P 5709442c75e74dd39217f2d0a08562b6: Deleting consensus metadata
I20260924 07:20:14.625126 2040 catalog_manager.cc:4841] TS 5709442c75e74dd39217f2d0a08562b6 (127.15.127.131:33607): tablet 1c1ce5bf777740fd95eeb25cac48e825 (table test-table [id=1fe19f71046d4639abeabd6aa835bdb8]) successfully deleted
I20260924 07:20:14.625728 2545 ts_tablet_manager.cc:1905] T 1c1ce5bf777740fd95eeb25cac48e825 P 36de5315f22c452fab25e261209feeee: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:20:14.626960 2040 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:45323 (error 108)
I20260924 07:20:14.627768 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2246
I20260924 07:20:14.628420 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.628060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:45323 (local address 127.0.0.1:40720)
0924 07:20:14.628242 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.628319 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:45323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260924 07:20:14.628593 2040 catalog_manager.cc:4584] TS a1d6299401c84db1a5ff5729a626f34e (127.15.127.129:45323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c1ce5bf777740fd95eeb25cac48e825: Network error: Client connection negotiation failed: client connection to 127.15.127.129:45323: connect: Connection refused (error 111)
I20260924 07:20:14.634338 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2377
I20260924 07:20:14.634912 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:14.634553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:34239 (local address 127.0.0.1:50820)
0924 07:20:14.634755 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:14.634815 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:34239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260924 07:20:14.639873 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2024
2026-09-24T07:20:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1084 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:14 dist-test-slave-zx46 krb5kdc[2551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:14 dist-test-slave-zx46 krb5kdc[2551](info): set up 2 sockets
Sep 24 07:20:14 dist-test-slave-zx46 krb5kdc[2551](info): commencing operation
krb5kdc: starting...
W20260924 07:20:15.664952 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:15 dist-test-slave-zx46 krb5kdc[2551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:15.783300 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:39403
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:39403
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:15.892911 2564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:15.893182 2564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:15.893230 2564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:15.897264 2564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:15.897323 2564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:15.897342 2564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:15.897359 2564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:15.897374 2564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:15.902618 2564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:39403
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:39403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:15.903599 2564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:15.904610 2564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:15.912884 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:15.912914 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:15.912935 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:15.913161 2564 server_base.cc:1034] running on GCE node
I20260924 07:20:15.913774 2564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:15.914836 2564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:15.915992 2564 hybrid_clock.cc:648] HybridClock initialized: now 1790234415915960 us; error 43 us; skew 500 ppm
Sep 24 07:20:15 dist-test-slave-zx46 krb5kdc[2551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234415, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:15.918792 2564 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:15.919661 2564 webserver.cc:460] Webserver started at http://127.15.127.190:37389/ using document root <none> and password file <none>
I20260924 07:20:15.920207 2564 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:15.920308 2564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:15.920527 2564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:15.922573 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "d607f26d6cd24f4fbb1023761902b6e9"
format_stamp: "Formatted at 2026-09-24 07:20:15 on dist-test-slave-zx46"
server_key: "35a7e8a5dcf7a884e45fed95ac940949"
server_key_iv: "df6771740408b57f4d1a5bd616fb2b90"
server_key_version: "encryptionkey@0"
I20260924 07:20:15.923125 2564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "d607f26d6cd24f4fbb1023761902b6e9"
format_stamp: "Formatted at 2026-09-24 07:20:15 on dist-test-slave-zx46"
server_key: "35a7e8a5dcf7a884e45fed95ac940949"
server_key_iv: "df6771740408b57f4d1a5bd616fb2b90"
server_key_version: "encryptionkey@0"
I20260924 07:20:15.926789 2564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:15.929136 2579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:15.930318 2564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:15.930462 2564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "d607f26d6cd24f4fbb1023761902b6e9"
format_stamp: "Formatted at 2026-09-24 07:20:15 on dist-test-slave-zx46"
server_key: "35a7e8a5dcf7a884e45fed95ac940949"
server_key_iv: "df6771740408b57f4d1a5bd616fb2b90"
server_key_version: "encryptionkey@0"
I20260924 07:20:15.930572 2564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:15.958217 2564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:15.961815 2564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:15.962033 2564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:15.968950 2564 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:39403
I20260924 07:20:15.968967 2631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:39403 every 8 connection(s)
I20260924 07:20:15.969910 2564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:15.972581 2632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:15.978030 2632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9: Bootstrap starting.
I20260924 07:20:15.979653 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2564
I20260924 07:20:15.979787 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:15.980077 15870 external_mini_cluster.cc:1428] Setting key 1f8dc28ff6dd82aece75c7bf86be2363
I20260924 07:20:15.980783 2632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:15.981666 2632 log.cc:824] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:15.983634 2632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9: No bootstrap required, opened a new log
I20260924 07:20:15.987154 2632 raft_consensus.cc:348] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d607f26d6cd24f4fbb1023761902b6e9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39403 } }
I20260924 07:20:15.987404 2632 raft_consensus.cc:374] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:15.987493 2632 raft_consensus.cc:729] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d607f26d6cd24f4fbb1023761902b6e9, State: Initialized, Role: FOLLOWER
I20260924 07:20:15.987958 2632 consensus_queue.cc:260] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d607f26d6cd24f4fbb1023761902b6e9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39403 } }
I20260924 07:20:15.988106 2632 raft_consensus.cc:388] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:15.988204 2632 raft_consensus.cc:482] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:15.988318 2632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:20:15 dist-test-slave-zx46 krb5kdc[2551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:15.989436 2632 raft_consensus.cc:504] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d607f26d6cd24f4fbb1023761902b6e9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39403 } }
I20260924 07:20:15.989778 2632 leader_election.cc:302] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d607f26d6cd24f4fbb1023761902b6e9; no voters:
I20260924 07:20:15.990135 2632 leader_election.cc:288] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:15.990238 2637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:15.990474 2637 raft_consensus.cc:686] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [term 1 LEADER]: Becoming Leader. State: Replica: d607f26d6cd24f4fbb1023761902b6e9, State: Running, Role: LEADER
I20260924 07:20:15.990824 2637 consensus_queue.cc:237] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d607f26d6cd24f4fbb1023761902b6e9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39403 } }
I20260924 07:20:15.991443 2632 sys_catalog.cc:564] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:15.992054 2637 sys_catalog.cc:455] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d607f26d6cd24f4fbb1023761902b6e9. Latest consensus state: current_term: 1 leader_uuid: "d607f26d6cd24f4fbb1023761902b6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d607f26d6cd24f4fbb1023761902b6e9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39403 } } }
I20260924 07:20:15.992169 2637 sys_catalog.cc:458] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:15.992542 2643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:15.992952 2638 sys_catalog.cc:455] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d607f26d6cd24f4fbb1023761902b6e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d607f26d6cd24f4fbb1023761902b6e9" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39403 } } }
I20260924 07:20:15.993163 2638 sys_catalog.cc:458] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:15.995818 2643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:15.996265 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:15.981203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46974 (local address 127.15.127.190:39403)
0924 07:20:15.981629 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:15.981644 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:15.981679 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:15.982543 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:15.982563 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:15.982882 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:15.983220 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:15.984054 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:15.985081 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:15.985948 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:15.986448 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:15.986660 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:15.989543 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:15.989550 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:15.989567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:15.989603 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:15.991724 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:15.992375 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:15.992382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:15.992387 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:15.992533 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:15.992905 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:15.992908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:15.992910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:15.993446 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:15.993687 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:15.994173 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 07:20:15.994627 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":129,"threads_started":1}
I20260924 07:20:16.004725 2643 catalog_manager.cc:1347] Generated new cluster ID: 59e55555c32f42eea90bc08dd1863fca
I20260924 07:20:16.004815 2643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:16.016047 2643 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:16.017621 2643 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:16.030493 2643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9: Generated new TSK 0
I20260924 07:20:16.031287 2643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.127.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
Entry for principal kudu/127.15.127.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab.
I20260924 07:20:16.071923 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39403
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:16.186760 2658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:16.186995 2658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:16.187071 2658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:16.190996 2658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:16.191123 2658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:16.191259 2658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:20:16.196604 2658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.129.keytab
--principal=kudu/127.15.127.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:16.197570 2658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:16.198554 2658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:16.207701 2663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:16.207715 2666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:16.207715 2664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:16.208076 2658 server_base.cc:1034] running on GCE node
I20260924 07:20:16.208417 2658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:16.208995 2658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:16.210222 2658 hybrid_clock.cc:648] HybridClock initialized: now 1790234416210194 us; error 38 us; skew 500 ppm
Sep 24 07:20:16 dist-test-slave-zx46 krb5kdc[2551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:16.212848 2658 init.cc:377] Logged in from keytab as kudu/127.15.127.129@KRBTEST.COM (short username kudu)
I20260924 07:20:16.213665 2658 webserver.cc:460] Webserver started at http://127.15.127.129:46419/ using document root <none> and password file <none>
I20260924 07:20:16.214321 2658 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:16.214411 2658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:16.214610 2658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:16.216509 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "baece382ef56446bafc17b5c8aaaa74f"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "4c731944e2979c15ca44f68adff29e68"
server_key_iv: "c16cb6b8df3ca5c698ee838acabf7c36"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.217140 2658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "baece382ef56446bafc17b5c8aaaa74f"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "4c731944e2979c15ca44f68adff29e68"
server_key_iv: "c16cb6b8df3ca5c698ee838acabf7c36"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.220664 2658 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 07:20:16.222962 2673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.224092 2658 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 07:20:16.224222 2658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "baece382ef56446bafc17b5c8aaaa74f"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "4c731944e2979c15ca44f68adff29e68"
server_key_iv: "c16cb6b8df3ca5c698ee838acabf7c36"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.224328 2658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:16.245796 2658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:16.248513 2658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:16.248724 2658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:16.249393 2658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:16.250212 2658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:16.250284 2658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.250353 2658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:16.250402 2658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.259562 2658 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:35831
I20260924 07:20:16.259573 2786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:35831 every 8 connection(s)
I20260924 07:20:16.260430 2658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:16.268510 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2658
I20260924 07:20:16.268661 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:16.268980 15870 external_mini_cluster.cc:1428] Setting key 6659336ec8bdb63fe06edca0f5d8b442
Sep 24 07:20:16 dist-test-slave-zx46 krb5kdc[2551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.129@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:16.274437 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.262153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:57671 (local address 127.15.127.190:39403)
0924 07:20:16.262290 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.262294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.263099 (+ 805us) server_negotiation.cc:407] Connection header received
0924 07:20:16.264366 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.264374 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.264419 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.264493 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:16.265702 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.266229 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.267295 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.267530 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.267607 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.270194 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:16.270198 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:16.270199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:16.270222 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:16.271931 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.272583 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.272586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.272587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.272661 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.273143 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.273145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.273146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.273331 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:16.273406 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.274005 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.274155 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 07:20:16.275039 2789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.262520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:39403 (local address 127.15.127.129:57671)
0924 07:20:16.262953 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.262994 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.264165 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.264650 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.264656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.265107 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:16.265574 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.265588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.266333 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.266335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.267192 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.267201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.267654 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.267656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.267798 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.268347 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:16.268369 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:16.270021 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:16.272048 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:16.272053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:16.272063 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:16.272479 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:16.272786 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:16.272790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:16.272794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:16.273008 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:16.273436 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:16.273443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:16.273748 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.273900 (+ 152us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.274146 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":116,"threads_started":1}
I20260924 07:20:16.276266 2787 heartbeater.cc:347] Connected to a master server at 127.15.127.190:39403
I20260924 07:20:16.276484 2787 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:16.276952 2787 heartbeater.cc:511] Master 127.15.127.190:39403 requested a full tablet report, sending...
I20260924 07:20:16.278424 2596 ts_manager.cc:194] Registered new tserver with Master: baece382ef56446bafc17b5c8aaaa74f (127.15.127.129:35831)
WARNING: no policy specified for kudu/127.15.127.130@KRBTEST.COM; defaulting to no policy
I20260924 07:20:16.279570 2596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.129@KRBTEST.COM'} at 127.15.127.129:57671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
Entry for principal kudu/127.15.127.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab.
I20260924 07:20:16.295902 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39403
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:16.402866 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:16.403115 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:16.403154 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:16.407064 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:16.407112 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:16.407203 2792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:20:16.412268 2792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.130.keytab
--principal=kudu/127.15.127.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:16.413261 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:16.414281 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:16.423503 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:16.423651 2792 server_base.cc:1034] running on GCE node
W20260924 07:20:16.423503 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:16.423712 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:16.424168 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:16.424767 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:16.425952 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790234416425921 us; error 40 us; skew 500 ppm
Sep 24 07:20:16 dist-test-slave-zx46 krb5kdc[2551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:16.428766 2792 init.cc:377] Logged in from keytab as kudu/127.15.127.130@KRBTEST.COM (short username kudu)
I20260924 07:20:16.429955 2792 webserver.cc:460] Webserver started at http://127.15.127.130:45901/ using document root <none> and password file <none>
I20260924 07:20:16.430632 2792 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:16.430721 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:16.430971 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:16.433561 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "3a5d68641b6d4382b6aea1c49228fe44"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "4e005b054a6ab0960ecceac61a1e8ecb"
server_key_iv: "f800144f56ef18938b7d1521e8449db3"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.434127 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "3a5d68641b6d4382b6aea1c49228fe44"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "4e005b054a6ab0960ecceac61a1e8ecb"
server_key_iv: "f800144f56ef18938b7d1521e8449db3"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.437769 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:16.440042 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.441349 2792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:16.441461 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "3a5d68641b6d4382b6aea1c49228fe44"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "4e005b054a6ab0960ecceac61a1e8ecb"
server_key_iv: "f800144f56ef18938b7d1521e8449db3"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.441569 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:16.458699 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:16.461937 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:16.462136 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:16.462705 2792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:16.463505 2792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:16.463559 2792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.463625 2792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:16.463663 2792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.472179 2792 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:44615
I20260924 07:20:16.472208 2920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:44615 every 8 connection(s)
I20260924 07:20:16.473073 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:20:16 dist-test-slave-zx46 krb5kdc[2551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.130@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:16.481993 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2792
I20260924 07:20:16.482160 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:16.482455 15870 external_mini_cluster.cc:1428] Setting key 642a712f60409abc24e6c0ec3034a4e1
I20260924 07:20:16.486095 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.474725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:33035 (local address 127.15.127.190:39403)
0924 07:20:16.474880 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.474884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.475661 (+ 777us) server_negotiation.cc:407] Connection header received
0924 07:20:16.476943 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.476946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.476994 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.477162 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:16.478281 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.478781 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.479869 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.480095 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.480172 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.482577 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:16.482581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:16.482584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:16.482616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:16.484162 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.484637 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.484642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.484644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.484721 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.485070 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.485073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.485074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.485244 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:16.485343 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.485811 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.485920 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:20:16.486913 2923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.475101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:39403 (local address 127.15.127.130:33035)
0924 07:20:16.475518 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.475558 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.476747 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.477328 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.477335 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.477671 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:16.478139 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.478153 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.478883 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.478885 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.479763 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.479771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.480225 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.480228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.480362 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.480877 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:16.480893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:16.482384 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:16.484271 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:16.484276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:16.484289 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:16.484532 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:16.484812 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:16.484815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:16.484817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:16.484951 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:16.485352 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:16.485360 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:16.485590 (+ 230us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.485761 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.486055 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260924 07:20:16.488040 2921 heartbeater.cc:347] Connected to a master server at 127.15.127.190:39403
I20260924 07:20:16.488287 2921 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:16.488904 2921 heartbeater.cc:511] Master 127.15.127.190:39403 requested a full tablet report, sending...
I20260924 07:20:16.490125 2596 ts_manager.cc:194] Registered new tserver with Master: 3a5d68641b6d4382b6aea1c49228fe44 (127.15.127.130:44615)
I20260924 07:20:16.490963 2596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.130@KRBTEST.COM'} at 127.15.127.130:33035
WARNING: no policy specified for kudu/127.15.127.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
Entry for principal kudu/127.15.127.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab.
I20260924 07:20:16.508178 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39403
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:34825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:16.618976 2926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:16.619252 2926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:16.619325 2926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:16.623359 2926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:16.623456 2926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:16.623546 2926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:20:16.628736 2926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:34825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.131.keytab
--principal=kudu/127.15.127.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:39403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:16.629724 2926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:16.630748 2926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:16.640211 2934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:16.640211 2932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:16.640224 2931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:16.640491 2926 server_base.cc:1034] running on GCE node
I20260924 07:20:16.641111 2926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:16.641763 2926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:16.642951 2926 hybrid_clock.cc:648] HybridClock initialized: now 1790234416642927 us; error 48 us; skew 500 ppm
Sep 24 07:20:16 dist-test-slave-zx46 krb5kdc[2551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:16.645834 2926 init.cc:377] Logged in from keytab as kudu/127.15.127.131@KRBTEST.COM (short username kudu)
I20260924 07:20:16.646708 2926 webserver.cc:460] Webserver started at http://127.15.127.131:43941/ using document root <none> and password file <none>
I20260924 07:20:16.647260 2926 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:16.647347 2926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:16.647545 2926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:16.649554 2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "a47ad911c47a4b0fbdc10b370d34eabd"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "10045c53b40e86885438e5ddd36a6b3e"
server_key_iv: "26db29cb49583439f80fc61dd6cb4933"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.650101 2926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "a47ad911c47a4b0fbdc10b370d34eabd"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "10045c53b40e86885438e5ddd36a6b3e"
server_key_iv: "26db29cb49583439f80fc61dd6cb4933"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.653626 2926 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:16.655920 2941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.657226 2926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:16.657352 2926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "a47ad911c47a4b0fbdc10b370d34eabd"
format_stamp: "Formatted at 2026-09-24 07:20:16 on dist-test-slave-zx46"
server_key: "10045c53b40e86885438e5ddd36a6b3e"
server_key_iv: "26db29cb49583439f80fc61dd6cb4933"
server_key_version: "encryptionkey@0"
I20260924 07:20:16.657460 2926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:16.674448 2926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:16.677421 2926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:16.677618 2926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:16.678205 2926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:16.679015 2926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:16.679086 2926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.679157 2926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:16.679199 2926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:16.687701 2926 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:44483
I20260924 07:20:16.687724 3054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:44483 every 8 connection(s)
I20260924 07:20:16.688540 2926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:16.694557 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 2926
I20260924 07:20:16.694694 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:16.694978 15870 external_mini_cluster.cc:1428] Setting key 3a2e76799e24aca27e12cff7f9404114
Sep 24 07:20:16 dist-test-slave-zx46 krb5kdc[2551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.131@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:16.702994 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.690416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:48849 (local address 127.15.127.190:39403)
0924 07:20:16.690559 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.690564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.691457 (+ 893us) server_negotiation.cc:407] Connection header received
0924 07:20:16.692951 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.692955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.693030 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.693133 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:16.694264 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.695183 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.696760 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.696968 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.697141 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.699556 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:16.699559 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:16.699561 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:16.699585 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:16.701152 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.701609 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.701612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.701613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.701703 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.701997 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.701999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.702000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.702192 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:16.702261 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.702779 (+ 518us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.702876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:20:16.703816 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.690815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:39403 (local address 127.15.127.131:48849)
0924 07:20:16.691275 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.691354 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.692715 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.693294 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.693301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.693632 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:16.694114 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.694135 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.695331 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.695335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.696652 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.696662 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.697170 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.697173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.697377 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.697950 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:16.697966 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:16.699391 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:16.701246 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:16.701251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:16.701261 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:16.701499 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:16.701796 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:16.701798 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:16.701799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:16.701924 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:16.702298 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:16.702303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:16.702553 (+ 250us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.702724 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.702965 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":107,"threads_started":1}
I20260924 07:20:16.704942 3055 heartbeater.cc:347] Connected to a master server at 127.15.127.190:39403
I20260924 07:20:16.705194 3055 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:16.705631 3055 heartbeater.cc:511] Master 127.15.127.190:39403 requested a full tablet report, sending...
I20260924 07:20:16.706537 2596 ts_manager.cc:194] Registered new tserver with Master: a47ad911c47a4b0fbdc10b370d34eabd (127.15.127.131:44483)
I20260924 07:20:16.707111 2596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.127.131@KRBTEST.COM'} at 127.15.127.131:48849
I20260924 07:20:16.708971 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:16.718644 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.710993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.15.127.190:39403)
0924 07:20:16.711157 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.711161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.711258 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:20:16.711349 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.711351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.711386 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.711449 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:16.712111 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.712610 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.713555 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.713786 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.713876 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.715213 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:16.715216 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:16.715218 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:16.715239 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:16.717088 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.717500 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.717502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.717504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.717576 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:16.717857 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:16.717859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:16.717860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:16.718055 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:16.718155 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.718368 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.718457 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 07:20:16.721360 2596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:16.724259 2596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:16.738348 3066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.732378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:44615 (local address 127.0.0.1:60526)
0924 07:20:16.733198 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.733245 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.733382 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.734198 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.734203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.734222 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:16.734324 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.734333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.735473 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.735479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.736665 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.736677 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.737802 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.737806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.737922 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.737962 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.738075 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.738182 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":1,"spinlock_wait_cycles":8960,"thread_start_us":128,"threads_started":1}
I20260924 07:20:16.738392 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.732580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.15.127.130:44615)
0924 07:20:16.733070 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.733082 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.733520 (+ 438us) server_negotiation.cc:407] Connection header received
0924 07:20:16.733655 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.733660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.733874 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.734020 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:16.734454 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.735349 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.736787 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.737672 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.737816 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.737963 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.738123 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.738219 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":129,"threads_started":1}
I20260924 07:20:16.738706 3067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.732517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:44483 (local address 127.0.0.1:33588)
0924 07:20:16.733197 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.733244 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.733380 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.734033 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.734038 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.734059 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:16.734202 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.734210 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.735679 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.735682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.736883 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.736892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.738136 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.738140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.738234 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.738256 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.738329 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.738395 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"spinlock_wait_cycles":9216,"thread_start_us":112,"threads_started":1}
I20260924 07:20:16.739208 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.733269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.15.127.131:44483)
0924 07:20:16.733657 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.733665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.733690 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:20:16.733747 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.733751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.733885 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.734041 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:16.734337 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.735504 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.737053 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.737835 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.737938 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.738073 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.738969 (+ 896us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.739077 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":138,"threads_started":1}
I20260924 07:20:16.739228 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.733994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:35831 (local address 127.0.0.1:43184)
0924 07:20:16.734311 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.734324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.734399 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.735161 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.735164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.735182 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:16.735285 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.735291 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.736700 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.736705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.737507 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.737516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.738843 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.738846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.738965 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.738987 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.739036 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.739098 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260924 07:20:16.740440 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.734407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.15.127.129:35831)
0924 07:20:16.734747 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.734761 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.734783 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:20:16.734851 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.734856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.735012 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.735151 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:16.735400 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.736559 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.737643 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.738670 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.740062 (+ 1392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.740223 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.740274 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.740340 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":81,"threads_started":1}
I20260924 07:20:16.741750 2855 tablet_service.cc:1467] Processing CreateTablet for tablet 744d7b7c34434bd0a630042cee6fd6e1 (DEFAULT_TABLE table=test-table [id=d404c457f2284238a6495d5ec86e1931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:16.741811 2989 tablet_service.cc:1467] Processing CreateTablet for tablet 744d7b7c34434bd0a630042cee6fd6e1 (DEFAULT_TABLE table=test-table [id=d404c457f2284238a6495d5ec86e1931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:16.742884 2989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744d7b7c34434bd0a630042cee6fd6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:16.742884 2855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744d7b7c34434bd0a630042cee6fd6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:16.742691 2721 tablet_service.cc:1467] Processing CreateTablet for tablet 744d7b7c34434bd0a630042cee6fd6e1 (DEFAULT_TABLE table=test-table [id=d404c457f2284238a6495d5ec86e1931]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:16.743777 2721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744d7b7c34434bd0a630042cee6fd6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:16.750007 3072 tablet_bootstrap.cc:492] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: Bootstrap starting.
I20260924 07:20:16.751274 3074 tablet_bootstrap.cc:492] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Bootstrap starting.
I20260924 07:20:16.751276 3073 tablet_bootstrap.cc:492] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Bootstrap starting.
I20260924 07:20:16.752216 3072 tablet_bootstrap.cc:654] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:16.753391 3072 log.cc:824] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:16.753476 3074 tablet_bootstrap.cc:654] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:16.753476 3073 tablet_bootstrap.cc:654] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:16.754863 3074 log.cc:824] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:16.754863 3073 log.cc:824] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:16.755738 3072 tablet_bootstrap.cc:492] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: No bootstrap required, opened a new log
I20260924 07:20:16.755957 3072 ts_tablet_manager.cc:1397] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 07:20:16.756825 3074 tablet_bootstrap.cc:492] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: No bootstrap required, opened a new log
I20260924 07:20:16.756825 3073 tablet_bootstrap.cc:492] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: No bootstrap required, opened a new log
I20260924 07:20:16.757036 3073 ts_tablet_manager.cc:1397] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:20:16.757036 3074 ts_tablet_manager.cc:1397] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260924 07:20:16.759016 3072 raft_consensus.cc:348] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.759270 3072 raft_consensus.cc:374] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:16.759325 3072 raft_consensus.cc:729] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baece382ef56446bafc17b5c8aaaa74f, State: Initialized, Role: FOLLOWER
I20260924 07:20:16.759778 3072 consensus_queue.cc:260] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.760488 2787 heartbeater.cc:503] Master 127.15.127.190:39403 was elected leader, sending a full tablet report...
I20260924 07:20:16.760847 3072 ts_tablet_manager.cc:1428] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 07:20:16.760912 3074 raft_consensus.cc:348] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.760911 3073 raft_consensus.cc:348] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.761183 3074 raft_consensus.cc:374] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:16.761193 3073 raft_consensus.cc:374] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:16.761237 3073 raft_consensus.cc:729] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5d68641b6d4382b6aea1c49228fe44, State: Initialized, Role: FOLLOWER
I20260924 07:20:16.761246 3074 raft_consensus.cc:729] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47ad911c47a4b0fbdc10b370d34eabd, State: Initialized, Role: FOLLOWER
W20260924 07:20:16.761531 2788 tablet.cc:2369] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:16.761710 3073 consensus_queue.cc:260] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.761777 3074 consensus_queue.cc:260] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.762687 3073 ts_tablet_manager.cc:1428] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:20:16.762698 3074 ts_tablet_manager.cc:1428] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:20:16.763027 3055 heartbeater.cc:503] Master 127.15.127.190:39403 was elected leader, sending a full tablet report...
I20260924 07:20:16.763027 2921 heartbeater.cc:503] Master 127.15.127.190:39403 was elected leader, sending a full tablet report...
I20260924 07:20:16.773699 3079 raft_consensus.cc:482] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:16.773831 3079 raft_consensus.cc:504] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.774860 3079 leader_election.cc:288] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a5d68641b6d4382b6aea1c49228fe44 (127.15.127.130:44615), baece382ef56446bafc17b5c8aaaa74f (127.15.127.129:35831)
I20260924 07:20:16.778313 3068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.775198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:33073 (local address 127.15.127.130:44615)
0924 07:20:16.775318 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.775322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.775337 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:20:16.775449 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.775452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.775492 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.775586 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:16.775862 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.776343 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.777380 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.777765 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.777909 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.777959 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.778132 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.778199 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:20:16.778314 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.775037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:44615 (local address 127.15.127.131:33073)
0924 07:20:16.775229 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.775246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.775330 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.775597 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.775600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.775617 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:16.775757 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.775762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.776452 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.776456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.777264 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.777273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.777901 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.777904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.778013 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.778030 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.778116 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.778192 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"spinlock_wait_cycles":44928}
I20260924 07:20:16.778564 3071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.775176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:54403 (local address 127.15.127.129:35831)
0924 07:20:16.775353 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.775359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.775562 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:20:16.775776 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.775780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.775825 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.775903 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:16.776156 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.776611 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.777657 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.778051 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.778187 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.778232 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.778398 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.778465 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 07:20:16.778576 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.775071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:35831 (local address 127.15.127.131:54403)
0924 07:20:16.775479 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.775493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:16.775674 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:16.775913 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:16.775916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:16.775930 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:16.776051 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.776056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.776720 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.776723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.777537 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:16.777545 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.778178 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.778181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:16.778274 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.778300 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 07:20:16.778384 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:20:16.778447 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":107,"threads_started":1}
I20260924 07:20:16.779047 2875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d7b7c34434bd0a630042cee6fd6e1" candidate_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5d68641b6d4382b6aea1c49228fe44" is_pre_election: true
I20260924 07:20:16.779080 2741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d7b7c34434bd0a630042cee6fd6e1" candidate_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baece382ef56446bafc17b5c8aaaa74f" is_pre_election: true
I20260924 07:20:16.779325 2875 raft_consensus.cc:2454] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47ad911c47a4b0fbdc10b370d34eabd in term 0.
I20260924 07:20:16.779325 2741 raft_consensus.cc:2454] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a47ad911c47a4b0fbdc10b370d34eabd in term 0.
I20260924 07:20:16.779779 2945 leader_election.cc:302] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47ad911c47a4b0fbdc10b370d34eabd, baece382ef56446bafc17b5c8aaaa74f; no voters:
I20260924 07:20:16.780043 3079 raft_consensus.cc:2781] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:16.780133 3079 raft_consensus.cc:482] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:16.780185 3079 raft_consensus.cc:3037] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:16.781306 3079 raft_consensus.cc:504] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.781625 3079 leader_election.cc:288] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [CANDIDATE]: Term 1 election: Requested vote from peers 3a5d68641b6d4382b6aea1c49228fe44 (127.15.127.130:44615), baece382ef56446bafc17b5c8aaaa74f (127.15.127.129:35831)
I20260924 07:20:16.781896 2875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d7b7c34434bd0a630042cee6fd6e1" candidate_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a5d68641b6d4382b6aea1c49228fe44"
I20260924 07:20:16.782039 2875 raft_consensus.cc:3037] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:16.782043 2741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d7b7c34434bd0a630042cee6fd6e1" candidate_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baece382ef56446bafc17b5c8aaaa74f"
I20260924 07:20:16.782148 2741 raft_consensus.cc:3037] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:16.783095 2875 raft_consensus.cc:2454] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47ad911c47a4b0fbdc10b370d34eabd in term 1.
I20260924 07:20:16.783133 2741 raft_consensus.cc:2454] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a47ad911c47a4b0fbdc10b370d34eabd in term 1.
I20260924 07:20:16.783401 2945 leader_election.cc:302] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a47ad911c47a4b0fbdc10b370d34eabd, baece382ef56446bafc17b5c8aaaa74f; no voters:
I20260924 07:20:16.783569 3079 raft_consensus.cc:2781] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:16.783808 3079 raft_consensus.cc:686] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 1 LEADER]: Becoming Leader. State: Replica: a47ad911c47a4b0fbdc10b370d34eabd, State: Running, Role: LEADER
I20260924 07:20:16.784137 3079 consensus_queue.cc:237] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } }
I20260924 07:20:16.787164 2595 catalog_manager.cc:5495] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd reported cstate change: term changed from 0 to 1, leader changed from <none> to a47ad911c47a4b0fbdc10b370d34eabd (127.15.127.131). New cstate: current_term: 1 leader_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47ad911c47a4b0fbdc10b370d34eabd" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 44483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260924 07:20:16.800289 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.797286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.15.127.131:44483)
0924 07:20:16.797407 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:16.797411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:16.797422 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:16.797504 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:16.797506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:16.797555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:16.797638 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:16.797832 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.798323 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.799089 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:16.799301 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:16.799372 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:16.799613 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:16.799994 (+ 381us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:16.800078 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:16.800144 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 07:20:16.800191 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:20:16.805742 2741 raft_consensus.cc:1261] T 744d7b7c34434bd0a630042cee6fd6e1 P baece382ef56446bafc17b5c8aaaa74f [term 1 FOLLOWER]: Refusing update from remote peer a47ad911c47a4b0fbdc10b370d34eabd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:16.805742 2875 raft_consensus.cc:1261] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 1 FOLLOWER]: Refusing update from remote peer a47ad911c47a4b0fbdc10b370d34eabd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:16.806382 3082 consensus_queue.cc:1035] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [LEADER]: Connected to new peer: Peer: permanent_uuid: "baece382ef56446bafc17b5c8aaaa74f" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:16.806595 3079 consensus_queue.cc:1035] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a5d68641b6d4382b6aea1c49228fe44" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:16.813035 3085 mvcc.cc:205] Tried to move back new op lower bound from 7332800171229802496 to 7332800171150635008. Current Snapshot: MvccSnapshot[applied={T|T < 7332800171229802496}]
I20260924 07:20:16.814122 3087 mvcc.cc:205] Tried to move back new op lower bound from 7332800171229802496 to 7332800171150635008. Current Snapshot: MvccSnapshot[applied={T|T < 7332800171229802496}]
I20260924 07:20:16.815659 3086 mvcc.cc:205] Tried to move back new op lower bound from 7332800171229802496 to 7332800171150635008. Current Snapshot: MvccSnapshot[applied={T|T < 7332800171229802496}]
I20260924 07:20:16.822525 2594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:16.822664 2594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:16.825139 2594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d607f26d6cd24f4fbb1023761902b6e9: Sending DeleteTablet for 3 replicas of tablet 744d7b7c34434bd0a630042cee6fd6e1
I20260924 07:20:16.825915 2989 tablet_service.cc:1514] Processing DeleteTablet for tablet 744d7b7c34434bd0a630042cee6fd6e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:16 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:33588
I20260924 07:20:16.825919 2855 tablet_service.cc:1514] Processing DeleteTablet for tablet 744d7b7c34434bd0a630042cee6fd6e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:16 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:60526
I20260924 07:20:16.826313 2721 tablet_service.cc:1514] Processing DeleteTablet for tablet 744d7b7c34434bd0a630042cee6fd6e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:16 UTC) from {username='kudu', principal='kudu/127.15.127.190@KRBTEST.COM'} at 127.0.0.1:43184
I20260924 07:20:16.826596 3094 tablet_replica.cc:330] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: stopping tablet replica
I20260924 07:20:16.826942 3094 raft_consensus.cc:2229] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:16.827067 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2658
I20260924 07:20:16.827358 3094 raft_consensus.cc:2258] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:16.827708 3095 tablet_replica.cc:330] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: stopping tablet replica
I20260924 07:20:16.828014 3095 raft_consensus.cc:2229] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:16.828302 3095 raft_consensus.cc:2258] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:16.830214 3094 ts_tablet_manager.cc:1905] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:16.832125 3095 ts_tablet_manager.cc:1905] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:20:16.832468 3094 ts_tablet_manager.cc:1918] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:16.832574 3094 log.cc:1196] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/744d7b7c34434bd0a630042cee6fd6e1
I20260924 07:20:16.832927 3094 ts_tablet_manager.cc:1939] T 744d7b7c34434bd0a630042cee6fd6e1 P a47ad911c47a4b0fbdc10b370d34eabd: Deleting consensus metadata
I20260924 07:20:16.834087 2581 catalog_manager.cc:4841] TS a47ad911c47a4b0fbdc10b370d34eabd (127.15.127.131:44483): tablet 744d7b7c34434bd0a630042cee6fd6e1 (table test-table [id=d404c457f2284238a6495d5ec86e1931]) successfully deleted
W20260924 07:20:16.835239 2583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:35831 (error 108)
I20260924 07:20:16.835484 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2792
I20260924 07:20:16.835584 3095 ts_tablet_manager.cc:1918] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:16.835668 3095 log.cc:1196] T 744d7b7c34434bd0a630042cee6fd6e1 P 3a5d68641b6d4382b6aea1c49228fe44: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790234163642002-15870-0/minicluster-data/ts-1/wal/wals/744d7b7c34434bd0a630042cee6fd6e1
I20260924 07:20:16.837638 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.837299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:35831 (local address 127.0.0.1:43192)
0924 07:20:16.837458 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.837540 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:35831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 07:20:16.837837 2583 catalog_manager.cc:4584] TS baece382ef56446bafc17b5c8aaaa74f (127.15.127.129:35831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 744d7b7c34434bd0a630042cee6fd6e1: Network error: Client connection negotiation failed: client connection to 127.15.127.129:35831: connect: Connection refused (error 111)
I20260924 07:20:16.842954 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2926
I20260924 07:20:16.843935 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:16.843227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:44615 (local address 127.0.0.1:60540)
0924 07:20:16.843777 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:16.843840 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468}
I20260924 07:20:16.848363 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 2564
2026-09-24T07:20:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2209 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T07:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:16Z Disabled control of system clock
I20260924 07:20:16.879042 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41537
--webserver_interface=127.15.127.190
--webserver_port=0
--builtin_ntp_servers=127.15.127.148:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:41537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:20:16.986802 3101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:16.987041 3101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:16.987074 3101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:16.990845 3101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:16.990888 3101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:16.990907 3101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:16.990922 3101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:16.990936 3101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:16.995406 3101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:41537
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:41537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:16.996326 3101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:16.997669 3101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:17.005662 3109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.005649 3106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.005672 3107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:17.005790 3101 server_base.cc:1034] running on GCE node
I20260924 07:20:17.006507 3101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:17.007509 3101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:17.008662 3101 hybrid_clock.cc:648] HybridClock initialized: now 1790234417008650 us; error 23 us; skew 500 ppm
I20260924 07:20:17.010437 3101 webserver.cc:460] Webserver started at http://127.15.127.190:44071/ using document root <none> and password file <none>
I20260924 07:20:17.010999 3101 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:17.011059 3101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:17.011294 3101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:17.013414 3101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "50cbe3c5eb7145b8819ef5a09108aaea"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "ef83bbe9640c47d840ea00b611658aa9"
server_key_iv: "3838c7e62e9f6457395b015e4d4ae628"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.013972 3101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "50cbe3c5eb7145b8819ef5a09108aaea"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "ef83bbe9640c47d840ea00b611658aa9"
server_key_iv: "3838c7e62e9f6457395b015e4d4ae628"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.017762 3101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:20:17.020004 3115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.021274 3101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:17.021391 3101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "50cbe3c5eb7145b8819ef5a09108aaea"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "ef83bbe9640c47d840ea00b611658aa9"
server_key_iv: "3838c7e62e9f6457395b015e4d4ae628"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.021492 3101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:17.034642 3101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:17.035228 3101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:17.035419 3101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:17.042140 3101 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:41537
I20260924 07:20:17.042156 3167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:41537 every 8 connection(s)
I20260924 07:20:17.043103 3101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:17.044431 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3101
I20260924 07:20:17.044580 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:17.044912 15870 external_mini_cluster.cc:1428] Setting key c5a991c34e266df26ac02a9c3b4fa083
I20260924 07:20:17.046659 3168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:17.052554 3168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: Bootstrap starting.
I20260924 07:20:17.053752 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.046275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.15.127.190:41537)
0924 07:20:17.046913 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.046928 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.046960 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:20:17.047550 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.047574 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.047927 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.048187 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:17.048491 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.049492 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.050633 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.051101 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.051219 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.052078 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:17.052083 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:17.052095 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:17.052453 (+ 358us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:17.052505 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:17.052524 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:17.052609 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.052800 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.053098 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":143,"threads_started":1}
I20260924 07:20:17.054862 3168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:17.055696 3168 log.cc:824] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:17.057533 3168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: No bootstrap required, opened a new log
I20260924 07:20:17.060755 3168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41537 } }
I20260924 07:20:17.060950 3168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:17.060983 3168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50cbe3c5eb7145b8819ef5a09108aaea, State: Initialized, Role: FOLLOWER
I20260924 07:20:17.061440 3168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41537 } }
I20260924 07:20:17.061605 3168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:17.061693 3168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:17.061791 3168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:17.062655 3168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41537 } }
I20260924 07:20:17.062929 3168 leader_election.cc:302] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50cbe3c5eb7145b8819ef5a09108aaea; no voters:
I20260924 07:20:17.063158 3168 leader_election.cc:288] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:17.063313 3173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:17.063563 3173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [term 1 LEADER]: Becoming Leader. State: Replica: 50cbe3c5eb7145b8819ef5a09108aaea, State: Running, Role: LEADER
I20260924 07:20:17.063926 3173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41537 } }
I20260924 07:20:17.064481 3168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:17.065538 3174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41537 } } }
I20260924 07:20:17.065661 3174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:17.066057 3175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50cbe3c5eb7145b8819ef5a09108aaea. Latest consensus state: current_term: 1 leader_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50cbe3c5eb7145b8819ef5a09108aaea" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 41537 } } }
I20260924 07:20:17.066157 3175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea [sys.catalog]: This master's current role is: LEADER
W20260924 07:20:17.069007 3188 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:17.069146 3188 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:17.069216 3182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:17.069845 3182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:17.075074 3182 catalog_manager.cc:1347] Generated new cluster ID: b6957121442845639f0ca92e500da3c5
I20260924 07:20:17.075134 3182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:17.086266 3182 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:17.087129 3182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:17.094348 3182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: Generated new TSK 0
I20260924 07:20:17.095422 3182 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:17.106921 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.129:0
--local_ip_for_outbound_sockets=127.15.127.129
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41537
--builtin_ntp_servers=127.15.127.148:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:20:17.214768 3192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:17.214998 3192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:17.215034 3192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:17.219002 3192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:17.219058 3192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:17.219108 3192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.129
I20260924 07:20:17.223826 3192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.127.129
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.129
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:17.224948 3192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:17.226397 3192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:17.235888 3197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.235936 3198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:17.235926 3192 server_base.cc:1034] running on GCE node
W20260924 07:20:17.235890 3200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:17.236471 3192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:17.237159 3192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:17.238349 3192 hybrid_clock.cc:648] HybridClock initialized: now 1790234417238314 us; error 45 us; skew 500 ppm
I20260924 07:20:17.240186 3192 webserver.cc:460] Webserver started at http://127.15.127.129:36219/ using document root <none> and password file <none>
I20260924 07:20:17.240748 3192 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:17.240839 3192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:17.241125 3192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:17.243211 3192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data/instance:
uuid: "0d11db3886fd4ee5a384eb478e9e6cb9"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "3775a88f7f6e6e78230ffe1efd9b0454"
server_key_iv: "b6654591489d2c06f4f1afcfb2f4c420"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.243768 3192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance:
uuid: "0d11db3886fd4ee5a384eb478e9e6cb9"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "3775a88f7f6e6e78230ffe1efd9b0454"
server_key_iv: "b6654591489d2c06f4f1afcfb2f4c420"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.247375 3192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:17.249752 3206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.251107 3192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:17.251267 3192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal
uuid: "0d11db3886fd4ee5a384eb478e9e6cb9"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "3775a88f7f6e6e78230ffe1efd9b0454"
server_key_iv: "b6654591489d2c06f4f1afcfb2f4c420"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.251392 3192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:17.273309 3192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:17.274025 3192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:17.274219 3192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:17.274797 3192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:17.275717 3192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:17.275786 3192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.275856 3192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:17.275907 3192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.284437 3192 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.129:37809
I20260924 07:20:17.284492 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.129:37809 every 8 connection(s)
I20260924 07:20:17.285341 3192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/data/info.pb
I20260924 07:20:17.293457 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3192
I20260924 07:20:17.293587 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-0/wal/instance
I20260924 07:20:17.293864 15870 external_mini_cluster.cc:1428] Setting key 1d5f82a5554444520925d434d7b12e7e
I20260924 07:20:17.294782 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.287159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.129:43505 (local address 127.15.127.190:41537)
0924 07:20:17.287295 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.287299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.288408 (+ 1109us) server_negotiation.cc:407] Connection header received
0924 07:20:17.289800 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.289803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.289883 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.289993 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:17.290880 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.291419 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.292570 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.292803 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.292915 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.294118 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:17.294122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:17.294125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:17.294137 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:17.294162 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:17.294169 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:17.294254 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.294581 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.294644 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:20:17.295598 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.287584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41537 (local address 127.15.127.129:43505)
0924 07:20:17.288238 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.288277 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.289582 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.290170 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.290177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.290260 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:17.290732 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.290744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.291554 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.291562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.292434 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.292443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.292955 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.292959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.293159 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.293833 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:17.293835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:17.293874 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:17.293934 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:17.294433 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:17.294441 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:17.294445 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.294571 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.294840 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260924 07:20:17.295771 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.130:0
--local_ip_for_outbound_sockets=127.15.127.130
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41537
--builtin_ntp_servers=127.15.127.148:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:20:17.296815 3320 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41537
I20260924 07:20:17.297096 3320 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:17.297581 3320 heartbeater.cc:511] Master 127.15.127.190:41537 requested a full tablet report, sending...
I20260924 07:20:17.299116 3132 ts_manager.cc:194] Registered new tserver with Master: 0d11db3886fd4ee5a384eb478e9e6cb9 (127.15.127.129:37809)
I20260924 07:20:17.300822 3132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.129:43505
W20260924 07:20:17.403458 3323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:17.403705 3323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:17.403740 3323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:17.407656 3323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:17.407702 3323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:17.407770 3323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.130
I20260924 07:20:17.412376 3323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.127.130
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.130
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:17.413350 3323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:17.414623 3323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:17.423911 3329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.423947 3328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.423969 3331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:17.423949 3323 server_base.cc:1034] running on GCE node
I20260924 07:20:17.424471 3323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:17.425163 3323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:17.426340 3323 hybrid_clock.cc:648] HybridClock initialized: now 1790234417426306 us; error 47 us; skew 500 ppm
I20260924 07:20:17.428067 3323 webserver.cc:460] Webserver started at http://127.15.127.130:41895/ using document root <none> and password file <none>
I20260924 07:20:17.428659 3323 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:17.428750 3323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:17.428975 3323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:17.430984 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data/instance:
uuid: "d7cbe22fa0114bee9506f55d5450be7f"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "253c2b5f79a2edecd7ba9d0cb3889b58"
server_key_iv: "3ce29ece9577c1e35dd6c6c4e9eaae17"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.431540 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance:
uuid: "d7cbe22fa0114bee9506f55d5450be7f"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "253c2b5f79a2edecd7ba9d0cb3889b58"
server_key_iv: "3ce29ece9577c1e35dd6c6c4e9eaae17"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.436524 3323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260924 07:20:17.438932 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.440136 3323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:17.440267 3323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal
uuid: "d7cbe22fa0114bee9506f55d5450be7f"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "253c2b5f79a2edecd7ba9d0cb3889b58"
server_key_iv: "3ce29ece9577c1e35dd6c6c4e9eaae17"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.440380 3323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:17.454924 3323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:17.455552 3323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:17.455750 3323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:17.456286 3323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:17.457228 3323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:17.457294 3323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.457360 3323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:17.457401 3323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.466516 3323 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.130:35915
I20260924 07:20:17.466542 3450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.130:35915 every 8 connection(s)
I20260924 07:20:17.467382 3323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/data/info.pb
I20260924 07:20:17.472600 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3323
I20260924 07:20:17.472755 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-1/wal/instance
I20260924 07:20:17.473151 15870 external_mini_cluster.cc:1428] Setting key 0f1601755388c7c6fd90b72699a2b172
I20260924 07:20:17.475173 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.127.131:0
--local_ip_for_outbound_sockets=127.15.127.131
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41537
--builtin_ntp_servers=127.15.127.148:45403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:20:17.477188 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.469145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.130:45229 (local address 127.15.127.190:41537)
0924 07:20:17.469290 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.469295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.470443 (+ 1148us) server_negotiation.cc:407] Connection header received
0924 07:20:17.471802 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.471805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.471851 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.471927 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:17.473035 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.473754 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.475157 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.475523 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.475641 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.476655 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:17.476658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:17.476661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:17.476673 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:17.476693 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:17.476708 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:17.476780 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.476972 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.477051 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:20:17.478034 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.469629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41537 (local address 127.15.127.130:45229)
0924 07:20:17.470307 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.470346 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.471619 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.472152 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.472162 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.472265 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:17.472859 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.472876 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.473886 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.473889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.475033 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.475042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.475675 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.475678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.475811 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.476416 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:17.476417 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:17.476455 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:17.476513 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:17.476793 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:17.476804 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:17.476809 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.476962 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.477288 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260924 07:20:17.479100 3451 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41537
I20260924 07:20:17.479387 3451 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:17.480023 3451 heartbeater.cc:511] Master 127.15.127.190:41537 requested a full tablet report, sending...
I20260924 07:20:17.481343 3132 ts_manager.cc:194] Registered new tserver with Master: d7cbe22fa0114bee9506f55d5450be7f (127.15.127.130:35915)
I20260924 07:20:17.482215 3132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.130:45229
W20260924 07:20:17.597606 3454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:17.597848 3454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:17.597884 3454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:17.601831 3454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:17.601876 3454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:17.601926 3454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.127.131
I20260924 07:20:17.606571 3454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:45403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.127.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.127.131
--webserver_port=0
--tserver_master_addrs=127.15.127.190:41537
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.127.131
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:17.607452 3454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:17.608747 3454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:17.617949 3462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.617961 3459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:17.617985 3460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:17.618310 3454 server_base.cc:1034] running on GCE node
I20260924 07:20:17.618665 3454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:17.619284 3454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:17.620465 3454 hybrid_clock.cc:648] HybridClock initialized: now 1790234417620430 us; error 47 us; skew 500 ppm
I20260924 07:20:17.622427 3454 webserver.cc:460] Webserver started at http://127.15.127.131:39527/ using document root <none> and password file <none>
I20260924 07:20:17.623000 3454 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:17.623090 3454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:17.623294 3454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:17.625504 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data/instance:
uuid: "4167fd82183949dc8ab89b7f48b9fc72"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "44e18a7f06f5a908e0e1647f07bcaaaf"
server_key_iv: "e11b767b9b6f32e17c6f4e2b3c7a9aa8"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.626056 3454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance:
uuid: "4167fd82183949dc8ab89b7f48b9fc72"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "44e18a7f06f5a908e0e1647f07bcaaaf"
server_key_iv: "e11b767b9b6f32e17c6f4e2b3c7a9aa8"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.629704 3454 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:17.632077 3468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.633388 3454 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:17.633518 3454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal
uuid: "4167fd82183949dc8ab89b7f48b9fc72"
format_stamp: "Formatted at 2026-09-24 07:20:17 on dist-test-slave-zx46"
server_key: "44e18a7f06f5a908e0e1647f07bcaaaf"
server_key_iv: "e11b767b9b6f32e17c6f4e2b3c7a9aa8"
server_key_version: "encryptionkey@0"
I20260924 07:20:17.633625 3454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:17.652789 3454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:17.653505 3454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:17.653707 3454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:17.654282 3454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:20:17.655315 3454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:20:17.655381 3454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.655468 3454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:20:17.655520 3454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:17.664386 3454 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.131:34743
I20260924 07:20:17.664443 3581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.131:34743 every 8 connection(s)
I20260924 07:20:17.665433 3454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/data/info.pb
I20260924 07:20:17.671741 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3454
I20260924 07:20:17.671838 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal/instance
I20260924 07:20:17.672111 15870 external_mini_cluster.cc:1428] Setting key 6ecba0552cdf8322cacb4e552d968085
I20260924 07:20:17.675280 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.667364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:41133 (local address 127.15.127.190:41537)
0924 07:20:17.667543 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.667547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.668616 (+ 1069us) server_negotiation.cc:407] Connection header received
0924 07:20:17.670149 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.670152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.670214 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.670354 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:17.671281 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.671876 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.673399 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.673689 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.673801 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.674894 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:17.674897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:17.674899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:17.674911 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:17.674933 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:17.674942 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:17.675015 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.675120 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.675170 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 07:20:17.676115 3584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.667751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:41537 (local address 127.15.127.131:41133)
0924 07:20:17.668460 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.668506 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.669892 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.670534 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.670541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.670628 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:17.671119 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.671132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.672037 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.672042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.673269 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.673279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.673831 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.673834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.673981 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.674651 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:20:17.674653 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:17.674694 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:20:17.674754 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:17.675010 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:17.675015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:17.675018 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.675098 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.675344 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":122,"threads_started":1}
I20260924 07:20:17.677326 3582 heartbeater.cc:347] Connected to a master server at 127.15.127.190:41537
I20260924 07:20:17.677578 3582 heartbeater.cc:464] Registering TS with master...
I20260924 07:20:17.678123 3582 heartbeater.cc:511] Master 127.15.127.190:41537 requested a full tablet report, sending...
I20260924 07:20:17.679126 3132 ts_manager.cc:194] Registered new tserver with Master: 4167fd82183949dc8ab89b7f48b9fc72 (127.15.127.131:34743)
I20260924 07:20:17.679687 3132 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.127.131:41133
I20260924 07:20:17.686146 15870 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:20:17.691566 3171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.688034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.15.127.190:41537)
0924 07:20:17.688151 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.688155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.688357 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:20:17.688466 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.688468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.688504 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.688569 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:17.688841 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.689390 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.690235 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.690431 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.690502 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.691204 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:17.691206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:17.691208 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:17.691218 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:17.691235 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:17.691243 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:17.691299 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.691427 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.691469 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:20:17.694222 3132 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35082:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:20:17.697263 3132 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:20:17.711897 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.706449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:35915 (local address 127.0.0.1:34142)
0924 07:20:17.706905 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.706920 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.707016 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.707769 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.707772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.707819 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:17.707995 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.708009 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.709502 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.709505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.710475 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.710504 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.711318 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.711323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.711430 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.711469 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.711607 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.711732 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":1,"thread_start_us":100,"threads_started":1}
I20260924 07:20:17.712007 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.706852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.15.127.130:35915)
0924 07:20:17.707286 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.707298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.707321 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:20:17.707396 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.707403 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.707613 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.707781 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:17.708144 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.709384 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.710611 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.711179 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.711332 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.711535 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.711728 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.711833 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":205,"threads_started":1}
I20260924 07:20:17.712596 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.706304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.15.127.129:37809)
0924 07:20:17.706784 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.706799 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.706834 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:17.706901 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.706905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.707062 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.707256 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:17.707623 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.708808 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.710549 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.711351 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.711466 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.711613 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.712326 (+ 713us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.712428 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":140,"threads_started":1}
I20260924 07:20:17.712834 3595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.705705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37809 (local address 127.0.0.1:47330)
0924 07:20:17.706401 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.706414 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.706500 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.707258 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.707264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.707293 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:17.707438 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.707445 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.708962 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.708966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.710412 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.710424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.711630 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.711634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.711809 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.711833 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.711900 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.711971 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":57,"thread_start_us":129,"threads_started":1}
I20260924 07:20:17.714141 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.705483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.131:34743 (local address 127.0.0.1:46442)
0924 07:20:17.705889 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.705944 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.706096 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.707790 (+ 1694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.707793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.707814 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:17.707953 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.707962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.710660 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.710663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.711797 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.711808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.713657 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.713660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.713817 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.713840 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.713934 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.714021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":43,"thread_start_us":79,"threads_started":1}
I20260924 07:20:17.715906 3385 tablet_service.cc:1467] Processing CreateTablet for tablet d51998661303468c9f6c595efc3831d7 (DEFAULT_TABLE table=test-table [id=ddfe43bcadd2445c9d8c329ded0972f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:17.715752 3254 tablet_service.cc:1467] Processing CreateTablet for tablet d51998661303468c9f6c595efc3831d7 (DEFAULT_TABLE table=test-table [id=ddfe43bcadd2445c9d8c329ded0972f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:17.716811 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51998661303468c9f6c595efc3831d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:17.716812 3254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51998661303468c9f6c595efc3831d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:17.717371 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.705679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46442 (local address 127.15.127.131:34743)
0924 07:20:17.707266 (+ 1587us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.707283 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.707307 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:20:17.707381 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.707386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.707623 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.708133 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:17.708279 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.710524 (+ 2245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.711935 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.713494 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.713652 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.717071 (+ 3419us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.717126 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.717220 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":137,"threads_started":1}
I20260924 07:20:17.720142 3516 tablet_service.cc:1467] Processing CreateTablet for tablet d51998661303468c9f6c595efc3831d7 (DEFAULT_TABLE table=test-table [id=ddfe43bcadd2445c9d8c329ded0972f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:20:17.721251 3516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d51998661303468c9f6c595efc3831d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:17.723165 3599 tablet_bootstrap.cc:492] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f: Bootstrap starting.
I20260924 07:20:17.724545 3600 tablet_bootstrap.cc:492] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: Bootstrap starting.
I20260924 07:20:17.725523 3599 tablet_bootstrap.cc:654] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:17.726361 3599 log.cc:824] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:17.726734 3600 tablet_bootstrap.cc:654] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:17.727545 3600 log.cc:824] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:17.727877 3602 tablet_bootstrap.cc:492] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Bootstrap starting.
I20260924 07:20:17.728493 3599 tablet_bootstrap.cc:492] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f: No bootstrap required, opened a new log
I20260924 07:20:17.728699 3599 ts_tablet_manager.cc:1397] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:20:17.729640 3600 tablet_bootstrap.cc:492] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: No bootstrap required, opened a new log
I20260924 07:20:17.729825 3600 ts_tablet_manager.cc:1397] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 07:20:17.730059 3602 tablet_bootstrap.cc:654] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:17.730891 3602 log.cc:824] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:17.732394 3599 raft_consensus.cc:348] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.732635 3599 raft_consensus.cc:374] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:17.732697 3599 raft_consensus.cc:729] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cbe22fa0114bee9506f55d5450be7f, State: Initialized, Role: FOLLOWER
I20260924 07:20:17.732920 3602 tablet_bootstrap.cc:492] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: No bootstrap required, opened a new log
I20260924 07:20:17.733155 3602 ts_tablet_manager.cc:1397] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260924 07:20:17.733196 3599 consensus_queue.cc:260] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.734212 3599 ts_tablet_manager.cc:1428] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260924 07:20:17.734053 3600 raft_consensus.cc:348] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.734311 3600 raft_consensus.cc:374] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:17.734361 3600 raft_consensus.cc:729] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d11db3886fd4ee5a384eb478e9e6cb9, State: Initialized, Role: FOLLOWER
I20260924 07:20:17.734575 3451 heartbeater.cc:503] Master 127.15.127.190:41537 was elected leader, sending a full tablet report...
I20260924 07:20:17.735890 3600 consensus_queue.cc:260] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.736557 3320 heartbeater.cc:503] Master 127.15.127.190:41537 was elected leader, sending a full tablet report...
I20260924 07:20:17.736889 3600 ts_tablet_manager.cc:1428] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 07:20:17.737290 3602 raft_consensus.cc:348] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.737550 3602 raft_consensus.cc:374] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:17.737619 3602 raft_consensus.cc:729] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4167fd82183949dc8ab89b7f48b9fc72, State: Initialized, Role: FOLLOWER
I20260924 07:20:17.738068 3602 consensus_queue.cc:260] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.738684 3582 heartbeater.cc:503] Master 127.15.127.190:41537 was elected leader, sending a full tablet report...
I20260924 07:20:17.739030 3602 ts_tablet_manager.cc:1428] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 07:20:17.784308 3607 raft_consensus.cc:482] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:20:17.784523 3607 raft_consensus.cc:504] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.785760 3607 leader_election.cc:288] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d11db3886fd4ee5a384eb478e9e6cb9 (127.15.127.129:37809), d7cbe22fa0114bee9506f55d5450be7f (127.15.127.130:35915)
W20260924 07:20:17.786548 3321 tablet.cc:2369] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:17.790802 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.786150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:51277 (local address 127.15.127.130:35915)
0924 07:20:17.786298 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.786303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.786566 (+ 263us) server_negotiation.cc:407] Connection header received
0924 07:20:17.786770 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.786774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.786822 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.786867 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:17.787347 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.788029 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.789499 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.790106 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.790244 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.790295 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.790590 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.790664 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 07:20:17.790802 3596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.786123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.127.131:56405 (local address 127.15.127.129:37809)
0924 07:20:17.786304 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.786309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.786324 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:20:17.786454 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.786457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.786503 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.786589 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:20:17.786901 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.787753 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.789499 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.790106 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.790243 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.790295 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.790590 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.790664 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 07:20:17.790813 3584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.785965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37809 (local address 127.15.127.131:56405)
0924 07:20:17.786213 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.786230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.786339 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.786634 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.786637 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.786654 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:17.786783 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.786792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.787871 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.787874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.789315 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.789326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.790247 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.790250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.790395 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.790470 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.790545 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.790664 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":144}
I20260924 07:20:17.790830 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.785987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:35915 (local address 127.15.127.131:51277)
0924 07:20:17.786483 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.786496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:17.786644 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:17.786997 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:17.786999 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:17.787014 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:20:17.787205 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.787215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.788131 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.788134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.789354 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:17.789364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.790252 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.790255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:17.790415 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.790465 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:20:17.790545 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:20:17.790660 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":115,"thread_start_us":171,"threads_started":1}
I20260924 07:20:17.791520 3405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51998661303468c9f6c595efc3831d7" candidate_uuid: "4167fd82183949dc8ab89b7f48b9fc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cbe22fa0114bee9506f55d5450be7f" is_pre_election: true
I20260924 07:20:17.791517 3274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51998661303468c9f6c595efc3831d7" candidate_uuid: "4167fd82183949dc8ab89b7f48b9fc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" is_pre_election: true
I20260924 07:20:17.791836 3274 raft_consensus.cc:2454] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4167fd82183949dc8ab89b7f48b9fc72 in term 0.
I20260924 07:20:17.791836 3405 raft_consensus.cc:2454] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4167fd82183949dc8ab89b7f48b9fc72 in term 0.
I20260924 07:20:17.792333 3471 leader_election.cc:302] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d11db3886fd4ee5a384eb478e9e6cb9, 4167fd82183949dc8ab89b7f48b9fc72; no voters:
I20260924 07:20:17.792625 3607 raft_consensus.cc:2781] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:20:17.792730 3607 raft_consensus.cc:482] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:20:17.792790 3607 raft_consensus.cc:3037] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:17.793820 3607 raft_consensus.cc:504] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.794165 3607 leader_election.cc:288] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [CANDIDATE]: Term 1 election: Requested vote from peers 0d11db3886fd4ee5a384eb478e9e6cb9 (127.15.127.129:37809), d7cbe22fa0114bee9506f55d5450be7f (127.15.127.130:35915)
I20260924 07:20:17.794484 3274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51998661303468c9f6c595efc3831d7" candidate_uuid: "4167fd82183949dc8ab89b7f48b9fc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9"
I20260924 07:20:17.794507 3405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d51998661303468c9f6c595efc3831d7" candidate_uuid: "4167fd82183949dc8ab89b7f48b9fc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cbe22fa0114bee9506f55d5450be7f"
I20260924 07:20:17.794615 3405 raft_consensus.cc:3037] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:17.794615 3274 raft_consensus.cc:3037] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:17.795877 3405 raft_consensus.cc:2454] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4167fd82183949dc8ab89b7f48b9fc72 in term 1.
I20260924 07:20:17.795878 3274 raft_consensus.cc:2454] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4167fd82183949dc8ab89b7f48b9fc72 in term 1.
I20260924 07:20:17.796195 3471 leader_election.cc:302] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d11db3886fd4ee5a384eb478e9e6cb9, 4167fd82183949dc8ab89b7f48b9fc72; no voters:
I20260924 07:20:17.796342 3607 raft_consensus.cc:2781] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:17.796531 3607 raft_consensus.cc:686] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 1 LEADER]: Becoming Leader. State: Replica: 4167fd82183949dc8ab89b7f48b9fc72, State: Running, Role: LEADER
I20260924 07:20:17.796877 3607 consensus_queue.cc:237] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } }
I20260924 07:20:17.799973 3132 catalog_manager.cc:5495] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4167fd82183949dc8ab89b7f48b9fc72 (127.15.127.131). New cstate: current_term: 1 leader_uuid: "4167fd82183949dc8ab89b7f48b9fc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4167fd82183949dc8ab89b7f48b9fc72" member_type: VOTER last_known_addr { host: "127.15.127.131" port: 34743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 } health_report { overall_health: UNKNOWN } } }
I20260924 07:20:17.833134 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.829563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46452 (local address 127.15.127.131:34743)
0924 07:20:17.829704 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:17.829709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:17.829724 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:20:17.829876 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:17.829878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:17.829936 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:17.830047 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:17.830286 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.830803 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.831688 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:17.831909 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:17.832015 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:17.832320 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:17.832737 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:17.832885 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:17.832925 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:20:17.832978 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:20:17.838934 3405 raft_consensus.cc:1261] T d51998661303468c9f6c595efc3831d7 P d7cbe22fa0114bee9506f55d5450be7f [term 1 FOLLOWER]: Refusing update from remote peer 4167fd82183949dc8ab89b7f48b9fc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:17.838935 3274 raft_consensus.cc:1261] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 1 FOLLOWER]: Refusing update from remote peer 4167fd82183949dc8ab89b7f48b9fc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:17.839576 3607 consensus_queue.cc:1035] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d11db3886fd4ee5a384eb478e9e6cb9" member_type: VOTER last_known_addr { host: "127.15.127.129" port: 37809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:17.839805 3609 consensus_queue.cc:1035] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cbe22fa0114bee9506f55d5450be7f" member_type: VOTER last_known_addr { host: "127.15.127.130" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:17.847095 3612 mvcc.cc:205] Tried to move back new op lower bound from 7332800175460855808 to 7332800175298924544. Current Snapshot: MvccSnapshot[applied={T|T < 7332800175460855808}]
I20260924 07:20:17.848147 3614 mvcc.cc:205] Tried to move back new op lower bound from 7332800175460855808 to 7332800175298924544. Current Snapshot: MvccSnapshot[applied={T|T < 7332800175460855808}]
I20260924 07:20:17.849476 3613 mvcc.cc:205] Tried to move back new op lower bound from 7332800175460855808 to 7332800175298924544. Current Snapshot: MvccSnapshot[applied={T|T < 7332800175460855808}]
I20260924 07:20:17.855609 3132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:17.855782 3132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35082:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:20:17.858153 3132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50cbe3c5eb7145b8819ef5a09108aaea: Sending DeleteTablet for 3 replicas of tablet d51998661303468c9f6c595efc3831d7
I20260924 07:20:17.858872 3254 tablet_service.cc:1514] Processing DeleteTablet for tablet d51998661303468c9f6c595efc3831d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:17 UTC) from {username='slave'} at 127.0.0.1:47330
I20260924 07:20:17.859102 3385 tablet_service.cc:1514] Processing DeleteTablet for tablet d51998661303468c9f6c595efc3831d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:17 UTC) from {username='slave'} at 127.0.0.1:34142
I20260924 07:20:17.859094 3516 tablet_service.cc:1514] Processing DeleteTablet for tablet d51998661303468c9f6c595efc3831d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:20:17 UTC) from {username='slave'} at 127.0.0.1:46442
I20260924 07:20:17.859390 3622 tablet_replica.cc:330] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9: stopping tablet replica
I20260924 07:20:17.859726 3622 raft_consensus.cc:2229] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:20:17.859830 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3192
I20260924 07:20:17.860028 3622 raft_consensus.cc:2258] T d51998661303468c9f6c595efc3831d7 P 0d11db3886fd4ee5a384eb478e9e6cb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:17.862401 3623 tablet_replica.cc:330] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: stopping tablet replica
I20260924 07:20:17.862782 3623 raft_consensus.cc:2229] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:20:17.863176 3623 raft_consensus.cc:2258] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:20:17.864637 3623 ts_tablet_manager.cc:1905] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:20:17.867522 3118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.127.129:37809 (error 108)
I20260924 07:20:17.868031 3623 ts_tablet_manager.cc:1918] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:20:17.868129 3623 log.cc:1196] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Deleting WAL directory at /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790234163642002-15870-0/minicluster-data/ts-2/wal/wals/d51998661303468c9f6c595efc3831d7
I20260924 07:20:17.868441 3623 ts_tablet_manager.cc:1939] T d51998661303468c9f6c595efc3831d7 P 4167fd82183949dc8ab89b7f48b9fc72: Deleting consensus metadata
I20260924 07:20:17.868522 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.868221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.129:37809 (local address 127.0.0.1:47338)
0924 07:20:17.868371 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.868441 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.129:37809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:20:17.868603 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3323
W20260924 07:20:17.868783 3118 catalog_manager.cc:4584] TS 0d11db3886fd4ee5a384eb478e9e6cb9 (127.15.127.129:37809): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d51998661303468c9f6c595efc3831d7: Network error: Client connection negotiation failed: client connection to 127.15.127.129:37809: connect: Connection refused (error 111)
I20260924 07:20:17.874004 3116 catalog_manager.cc:4841] TS 4167fd82183949dc8ab89b7f48b9fc72 (127.15.127.131:34743): tablet d51998661303468c9f6c595efc3831d7 (table test-table [id=ddfe43bcadd2445c9d8c329ded0972f8]) successfully deleted
I20260924 07:20:17.874631 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3454
I20260924 07:20:17.875649 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:17.874785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.130:35915 (local address 127.0.0.1:34146)
0924 07:20:17.875496 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:17.875558 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.130:35915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616}
I20260924 07:20:17.879496 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3101
2026-09-24T07:20:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1029 ms)
[----------] 6 tests from EncryptionPolicyTest (10145 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:17 dist-test-slave-zx46 krb5kdc[3628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:17 dist-test-slave-zx46 krb5kdc[3628](info): set up 2 sockets
Sep 24 07:20:17 dist-test-slave-zx46 krb5kdc[3628](info): commencing operation
krb5kdc: starting...
W20260924 07:20:18.921370 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:18 dist-test-slave-zx46 krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:19.039608 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34879
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:19.145612 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:19.145864 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:19.145902 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:19.149596 3641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:19.149641 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:19.149658 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:19.149679 3641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:19.149736 3641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:19.154791 3641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34879
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:19.155786 3641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:19.156793 3641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:19.165128 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:19.165201 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:19.165231 3641 server_base.cc:1034] running on GCE node
W20260924 07:20:19.165318 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:19.165884 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:19.166923 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:19.168073 3641 hybrid_clock.cc:648] HybridClock initialized: now 1790234419168050 us; error 37 us; skew 500 ppm
Sep 24 07:20:19 dist-test-slave-zx46 krb5kdc[3628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234419, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:19.170864 3641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:19.171738 3641 webserver.cc:460] Webserver started at http://127.0.0.1:43989/ using document root <none> and password file <none>
I20260924 07:20:19.172276 3641 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:19.172367 3641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:19.172592 3641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:19.174769 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "190074c6a7314ab3a63ff1a2818ceb8e"
format_stamp: "Formatted at 2026-09-24 07:20:19 on dist-test-slave-zx46"
server_key: "81ffa7f7b455d2ffa81187894d70badc"
server_key_iv: "1b41cb56cbd24c57ada4a0667f14e392"
server_key_version: "encryptionkey@0"
I20260924 07:20:19.175340 3641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "190074c6a7314ab3a63ff1a2818ceb8e"
format_stamp: "Formatted at 2026-09-24 07:20:19 on dist-test-slave-zx46"
server_key: "81ffa7f7b455d2ffa81187894d70badc"
server_key_iv: "1b41cb56cbd24c57ada4a0667f14e392"
server_key_version: "encryptionkey@0"
I20260924 07:20:19.178900 3641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 07:20:19.181152 3656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:19.182260 3641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:19.182410 3641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "190074c6a7314ab3a63ff1a2818ceb8e"
format_stamp: "Formatted at 2026-09-24 07:20:19 on dist-test-slave-zx46"
server_key: "81ffa7f7b455d2ffa81187894d70badc"
server_key_iv: "1b41cb56cbd24c57ada4a0667f14e392"
server_key_version: "encryptionkey@0"
I20260924 07:20:19.182515 3641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:19.193145 3641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:19.196337 3641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:19.196554 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:19.203759 3641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34879
I20260924 07:20:19.203778 3708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34879 every 8 connection(s)
I20260924 07:20:19.204690 3641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:19.205425 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3641
I20260924 07:20:19.205569 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:19.205904 15870 external_mini_cluster.cc:1428] Setting key abd58ddd9e7ff8d5823bada3675a90f6
I20260924 07:20:19.207870 3709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:19.213340 3709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e: Bootstrap starting.
Sep 24 07:20:19 dist-test-slave-zx46 krb5kdc[3628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:20:19.215736 3709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:19.216452 3709 log.cc:824] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:19.218380 3709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e: No bootstrap required, opened a new log
I20260924 07:20:19.220338 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:19.207277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37323 (local address 127.0.0.1:34879)
0924 07:20:19.207709 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:19.207724 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:19.207759 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:19.208448 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:19.208466 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:19.208800 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:19.209156 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:19.209960 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:19.210954 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:19.211821 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:19.212293 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:19.212473 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:19.215313 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:19.215318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:19.215331 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:19.215357 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:19.217406 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:19.217926 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:19.217938 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:19.217943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:19.218048 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:19.218392 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:19.218395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:19.218397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:19.218836 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:19.219060 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:19.219393 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 07:20:19.219627 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":143,"threads_started":1}
I20260924 07:20:19.222479 3709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34879 } }
I20260924 07:20:19.222738 3709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:19.222797 3709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190074c6a7314ab3a63ff1a2818ceb8e, State: Initialized, Role: FOLLOWER
I20260924 07:20:19.223297 3709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34879 } }
I20260924 07:20:19.223459 3709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:19.223553 3709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:19.223567 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:19.223682 3709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:19.224582 3709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34879 } }
I20260924 07:20:19.224972 3709 leader_election.cc:302] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 190074c6a7314ab3a63ff1a2818ceb8e; no voters:
I20260924 07:20:19.225337 3709 leader_election.cc:288] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:19.225435 3718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:19.225790 3718 raft_consensus.cc:686] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [term 1 LEADER]: Becoming Leader. State: Replica: 190074c6a7314ab3a63ff1a2818ceb8e, State: Running, Role: LEADER
I20260924 07:20:19.226080 3718 consensus_queue.cc:237] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34879 } }
I20260924 07:20:19.226649 3709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:19.227180 3718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 190074c6a7314ab3a63ff1a2818ceb8e. Latest consensus state: current_term: 1 leader_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34879 } } }
I20260924 07:20:19.227321 3718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:19.227644 3721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "190074c6a7314ab3a63ff1a2818ceb8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34879 } } }
I20260924 07:20:19.227723 3721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:19.227777 3724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:19.230890 3724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:19.235330 3712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:19.225514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43941 (local address 127.0.0.1:34879)
0924 07:20:19.225623 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:19.225629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:19.225997 (+ 368us) server_negotiation.cc:407] Connection header received
0924 07:20:19.226147 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:19.226150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:19.226204 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:19.226253 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:19.227211 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:19.228041 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:19.229337 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:19.229663 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:19.229739 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:19.231601 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:19.231605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:19.231608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:19.231633 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:19.233369 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:19.233922 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:19.233926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:19.233928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:19.234011 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:19.234369 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:19.234372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:19.234374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:19.234573 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:19.234714 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:19.235064 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 07:20:19.235164 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":37}
I20260924 07:20:19.237124 3724 catalog_manager.cc:1347] Generated new cluster ID: 493d5cdcdd654bcfbf8bda04f1d15667
I20260924 07:20:19.237190 3724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:19.248003 3724 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:19.249264 3724 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:19.264096 3724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 190074c6a7314ab3a63ff1a2818ceb8e: Generated new TSK 0
I20260924 07:20:19.265031 3724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:19.274403 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3641
W20260924 07:20:19.279758 3626 connection.cc:535] client connection to 127.0.0.1:34879 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34879): Connection reset by peer (error 104)
2026-09-24T07:20:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:19 dist-test-slave-zx46 krb5kdc[3740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:19 dist-test-slave-zx46 krb5kdc[3740](info): set up 2 sockets
Sep 24 07:20:19 dist-test-slave-zx46 krb5kdc[3740](info): commencing operation
krb5kdc: starting...
W20260924 07:20:20.301249 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:20 dist-test-slave-zx46 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:20.420032 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35865
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:20.533237 3753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:20.533502 3753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:20.533555 3753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:20.537539 3753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:20.537598 3753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:20.537617 3753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:20.537633 3753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:20.537648 3753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:20.542760 3753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35865
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:20.543707 3753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:20.544651 3753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:20.552979 3761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:20.553018 3758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:20.552979 3759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:20.553375 3753 server_base.cc:1034] running on GCE node
I20260924 07:20:20.553946 3753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:20.555018 3753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:20.556212 3753 hybrid_clock.cc:648] HybridClock initialized: now 1790234420556196 us; error 52 us; skew 500 ppm
Sep 24 07:20:20 dist-test-slave-zx46 krb5kdc[3740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:20.558923 3753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:20.559753 3753 webserver.cc:460] Webserver started at http://127.0.0.1:41661/ using document root <none> and password file <none>
I20260924 07:20:20.560256 3753 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:20.560328 3753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:20.560482 3753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:20.562469 3753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "7923fb5d89a4409099b67fc5dc9224e0"
format_stamp: "Formatted at 2026-09-24 07:20:20 on dist-test-slave-zx46"
server_key: "408cbb89a6af4f7ab76aafafcc2ea47d"
server_key_iv: "d8d0397a4905d24c97455c88dcffebf8"
server_key_version: "encryptionkey@0"
I20260924 07:20:20.562964 3753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "7923fb5d89a4409099b67fc5dc9224e0"
format_stamp: "Formatted at 2026-09-24 07:20:20 on dist-test-slave-zx46"
server_key: "408cbb89a6af4f7ab76aafafcc2ea47d"
server_key_iv: "d8d0397a4905d24c97455c88dcffebf8"
server_key_version: "encryptionkey@0"
I20260924 07:20:20.566540 3753 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 07:20:20.568809 3768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:20.570057 3753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:20:20.570156 3753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "7923fb5d89a4409099b67fc5dc9224e0"
format_stamp: "Formatted at 2026-09-24 07:20:20 on dist-test-slave-zx46"
server_key: "408cbb89a6af4f7ab76aafafcc2ea47d"
server_key_iv: "d8d0397a4905d24c97455c88dcffebf8"
server_key_version: "encryptionkey@0"
I20260924 07:20:20.570245 3753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:20.608494 3753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:20.611333 3753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:20.611487 3753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:20.618786 3753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35865
I20260924 07:20:20.618804 3820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35865 every 8 connection(s)
I20260924 07:20:20.619778 3753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:20.622617 3821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:20.626482 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3753
I20260924 07:20:20.626618 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:20.626940 15870 external_mini_cluster.cc:1428] Setting key 6aa691a38c8565509d408585e6048e57
I20260924 07:20:20.628794 3821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0: Bootstrap starting.
I20260924 07:20:20.631081 3821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:20.631829 3821 log.cc:824] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:20.633805 3821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0: No bootstrap required, opened a new log
I20260924 07:20:20.636121 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:20.628425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59139 (local address 127.0.0.1:35865)
0924 07:20:20.628884 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:20.628900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:20.628972 (+ 72us) server_negotiation.cc:407] Connection header received
0924 07:20:20.629689 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:20.629707 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:20.630045 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:20.630405 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:20.630839 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:20.631877 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:20.632789 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:20.633328 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:20.633510 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:20.634486 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:20.634491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:20.634504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:20.634888 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:20.634941 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:20.634964 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:20.635105 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:20.635276 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 07:20:20.635420 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":157,"threads_started":1}
I20260924 07:20:20.637750 3821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fb5d89a4409099b67fc5dc9224e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 07:20:20.638007 3821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:20.638069 3821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7923fb5d89a4409099b67fc5dc9224e0, State: Initialized, Role: FOLLOWER
I20260924 07:20:20.638625 3821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fb5d89a4409099b67fc5dc9224e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 07:20:20.638803 3821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:20.638892 3821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:20.638994 3821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:20.639367 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:20.640066 3821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fb5d89a4409099b67fc5dc9224e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 07:20:20.640427 3821 leader_election.cc:302] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7923fb5d89a4409099b67fc5dc9224e0; no voters:
I20260924 07:20:20.640777 3821 leader_election.cc:288] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:20.640844 3829 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:20.641064 3829 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [term 1 LEADER]: Becoming Leader. State: Replica: 7923fb5d89a4409099b67fc5dc9224e0, State: Running, Role: LEADER
I20260924 07:20:20.641444 3829 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fb5d89a4409099b67fc5dc9224e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } }
I20260924 07:20:20.642154 3821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:20.643220 3832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7923fb5d89a4409099b67fc5dc9224e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fb5d89a4409099b67fc5dc9224e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } } }
I20260924 07:20:20.643435 3831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7923fb5d89a4409099b67fc5dc9224e0. Latest consensus state: current_term: 1 leader_uuid: "7923fb5d89a4409099b67fc5dc9224e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7923fb5d89a4409099b67fc5dc9224e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35865 } } }
I20260924 07:20:20.643551 3832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:20.643900 3831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:20.646682 3824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:20.641819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37187 (local address 127.0.0.1:35865)
0924 07:20:20.641949 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:20.641953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:20.642108 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:20:20.642275 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:20.642280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:20.642332 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:20.642421 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:20.642674 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:20.643495 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:20.644702 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:20.645074 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:20.645183 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:20.646184 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:20.646188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:20.646191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:20.646204 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:20.646231 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:20.646244 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:20.646331 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:20.646509 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:20:20.646544 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 07:20:20.647578 3846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:20.647684 3846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:20.647838 3840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:20.648730 3840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:20.654682 3840 catalog_manager.cc:1347] Generated new cluster ID: 5b2a60c48ebf4cd4a0fbc9e4695726e3
I20260924 07:20:20.654759 3840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:20.679749 3840 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:20.680769 3840 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:20.692754 3840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7923fb5d89a4409099b67fc5dc9224e0: Generated new TSK 0
I20260924 07:20:20.693651 3840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:20.701599 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3753
W20260924 07:20:20.707011 3738 connection.cc:535] client connection to 127.0.0.1:35865 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35865): Connection reset by peer (error 104)
2026-09-24T07:20:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:20 dist-test-slave-zx46 krb5kdc[3852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:20 dist-test-slave-zx46 krb5kdc[3852](info): set up 2 sockets
Sep 24 07:20:20 dist-test-slave-zx46 krb5kdc[3852](info): commencing operation
krb5kdc: starting...
W20260924 07:20:21.725687 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:21 dist-test-slave-zx46 krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:21.846351 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:21.953858 3865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:21.954118 3865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:21.954213 3865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:21.958249 3865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:21.958310 3865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:21.958329 3865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:21.958376 3865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:21.958413 3865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:21.963652 3865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:21.964699 3865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:21.965749 3865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:20:21.974231 3865 server_base.cc:1034] running on GCE node
W20260924 07:20:21.974181 3871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:21.974169 3873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:21.974169 3870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:21.975013 3865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:21.976022 3865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:21.977178 3865 hybrid_clock.cc:648] HybridClock initialized: now 1790234421977150 us; error 43 us; skew 500 ppm
Sep 24 07:20:21 dist-test-slave-zx46 krb5kdc[3852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:21.979804 3865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:21.980686 3865 webserver.cc:460] Webserver started at http://127.0.0.1:42879/ using document root <none> and password file <none>
I20260924 07:20:21.981330 3865 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:21.981392 3865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:21.981681 3865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:21.983682 3865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "8716cb3e284f455f95ce10691ecdf7ca"
format_stamp: "Formatted at 2026-09-24 07:20:21 on dist-test-slave-zx46"
server_key: "1668b357910dbbc039f2dc8fbc8719e6"
server_key_iv: "4b4ab641b9571ff02e9156aa0b5dceb1"
server_key_version: "encryptionkey@0"
I20260924 07:20:21.984272 3865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "8716cb3e284f455f95ce10691ecdf7ca"
format_stamp: "Formatted at 2026-09-24 07:20:21 on dist-test-slave-zx46"
server_key: "1668b357910dbbc039f2dc8fbc8719e6"
server_key_iv: "4b4ab641b9571ff02e9156aa0b5dceb1"
server_key_version: "encryptionkey@0"
I20260924 07:20:21.987999 3865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 07:20:21.990386 3880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:21.991580 3865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:21.991719 3865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "8716cb3e284f455f95ce10691ecdf7ca"
format_stamp: "Formatted at 2026-09-24 07:20:21 on dist-test-slave-zx46"
server_key: "1668b357910dbbc039f2dc8fbc8719e6"
server_key_iv: "4b4ab641b9571ff02e9156aa0b5dceb1"
server_key_version: "encryptionkey@0"
I20260924 07:20:21.991829 3865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:21.996268 3865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:21.998996 3865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:21.999190 3865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:22.006049 3865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34481
I20260924 07:20:22.006076 3932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34481 every 8 connection(s)
I20260924 07:20:22.006946 3865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:22.009671 3933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:22.012207 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3865
I20260924 07:20:22.012373 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:22.012665 15870 external_mini_cluster.cc:1428] Setting key 3c42997dbb2791ea13d8f6a596ad33cc
I20260924 07:20:22.015561 3933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca: Bootstrap starting.
I20260924 07:20:22.018762 3933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:22.019200 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:22.013980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44769 (local address 127.0.0.1:34481)
0924 07:20:22.014372 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:22.014386 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:22.014420 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:22.014986 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:22.015028 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:22.015449 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:22.015889 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:22.017620 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:22.017625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:22.017640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:22.017671 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:22.017738 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:22.017782 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:22.017911 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:22.017982 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:20:22.018180 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":134,"threads_started":1}
I20260924 07:20:22.019712 3933 log.cc:824] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:22.021966 3933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca: No bootstrap required, opened a new log
I20260924 07:20:22.022342 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:22.026084 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:22.024518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58653 (local address 127.0.0.1:34481)
0924 07:20:22.024637 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:22.024641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:22.024819 (+ 178us) server_negotiation.cc:407] Connection header received
0924 07:20:22.024915 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:22.024918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:22.024951 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:22.025050 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:22.025752 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:22.025757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:22.025761 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:22.025772 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:22.025798 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:22.025810 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:22.025906 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:22.025939 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 07:20:22.025973 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:20:22.026618 3933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8716cb3e284f455f95ce10691ecdf7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260924 07:20:22.026860 3933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:22.026918 3933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8716cb3e284f455f95ce10691ecdf7ca, State: Initialized, Role: FOLLOWER
I20260924 07:20:22.027385 3933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8716cb3e284f455f95ce10691ecdf7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260924 07:20:22.027535 3933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:22.027604 3933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:22.027700 3933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:22.028566 3933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8716cb3e284f455f95ce10691ecdf7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260924 07:20:22.028899 3933 leader_election.cc:302] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8716cb3e284f455f95ce10691ecdf7ca; no voters:
I20260924 07:20:22.029206 3933 leader_election.cc:288] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:22.029323 3943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:22.029503 3943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [term 1 LEADER]: Becoming Leader. State: Replica: 8716cb3e284f455f95ce10691ecdf7ca, State: Running, Role: LEADER
I20260924 07:20:22.029826 3943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8716cb3e284f455f95ce10691ecdf7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } }
I20260924 07:20:22.030328 3933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:22.031378 3944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8716cb3e284f455f95ce10691ecdf7ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8716cb3e284f455f95ce10691ecdf7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } } }
I20260924 07:20:22.031519 3944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:22.031834 3945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8716cb3e284f455f95ce10691ecdf7ca. Latest consensus state: current_term: 1 leader_uuid: "8716cb3e284f455f95ce10691ecdf7ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8716cb3e284f455f95ce10691ecdf7ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34481 } } }
I20260924 07:20:22.031919 3952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:22.031919 3945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:22.034873 3952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:22.040601 3952 catalog_manager.cc:1347] Generated new cluster ID: 9a1ecb5cd17843e6b9c3350fc57f87de
I20260924 07:20:22.040721 3952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:22.050809 3952 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:22.051100 3952 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:22.068022 3952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8716cb3e284f455f95ce10691ecdf7ca: Generated new TSK 0
I20260924 07:20:22.068760 3952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:22.075169 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3865
2026-09-24T07:20:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:22 dist-test-slave-zx46 krb5kdc[3964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:22 dist-test-slave-zx46 krb5kdc[3964](info): set up 2 sockets
Sep 24 07:20:22 dist-test-slave-zx46 krb5kdc[3964](info): commencing operation
krb5kdc: starting...
W20260924 07:20:23.097275 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:23 dist-test-slave-zx46 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:23.217799 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:23.325475 3977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:23.325768 3977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:23.325862 3977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:23.329842 3977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:23.329932 3977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:23.329969 3977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:23.330039 3977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:23.330083 3977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:23.335304 3977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:23.336339 3977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:23.337373 3977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:23.345817 3983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:23.345808 3985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:23.345875 3977 server_base.cc:1034] running on GCE node
W20260924 07:20:23.345808 3982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:23.346611 3977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:23.347645 3977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:23.348832 3977 hybrid_clock.cc:648] HybridClock initialized: now 1790234423348819 us; error 24 us; skew 500 ppm
Sep 24 07:20:23 dist-test-slave-zx46 krb5kdc[3964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234423, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:23.359858 3977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:23.360780 3977 webserver.cc:460] Webserver started at http://127.0.0.1:39103/ using document root <none> and password file <none>
I20260924 07:20:23.361370 3977 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:23.361433 3977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:23.361604 3977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:23.363579 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "0f7623b940634227932d929393238a15"
format_stamp: "Formatted at 2026-09-24 07:20:23 on dist-test-slave-zx46"
server_key: "528be47137f25ac94a0e8eb1ced6ac74"
server_key_iv: "0a1be19c3a1f63843f693ab7662cab33"
server_key_version: "encryptionkey@0"
I20260924 07:20:23.364085 3977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "0f7623b940634227932d929393238a15"
format_stamp: "Formatted at 2026-09-24 07:20:23 on dist-test-slave-zx46"
server_key: "528be47137f25ac94a0e8eb1ced6ac74"
server_key_iv: "0a1be19c3a1f63843f693ab7662cab33"
server_key_version: "encryptionkey@0"
I20260924 07:20:23.367645 3977 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260924 07:20:23.369941 3992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:23.371131 3977 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:23.371245 3977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "0f7623b940634227932d929393238a15"
format_stamp: "Formatted at 2026-09-24 07:20:23 on dist-test-slave-zx46"
server_key: "528be47137f25ac94a0e8eb1ced6ac74"
server_key_iv: "0a1be19c3a1f63843f693ab7662cab33"
server_key_version: "encryptionkey@0"
I20260924 07:20:23.371327 3977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:23.393319 3977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:23.396613 3977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:23.396765 3977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:23.403529 3977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40231
I20260924 07:20:23.403553 4044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40231 every 8 connection(s)
I20260924 07:20:23.404479 3977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:23.407354 4045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:23.412838 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15: Bootstrap starting.
I20260924 07:20:23.413993 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 3977
I20260924 07:20:23.414104 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:23.414422 15870 external_mini_cluster.cc:1428] Setting key 78a1ce5b1dd870e36024a49be4fc865e
I20260924 07:20:23.415454 4045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:23.416255 4045 log.cc:824] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:23.418445 4045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15: No bootstrap required, opened a new log
I20260924 07:20:23.422673 4045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7623b940634227932d929393238a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260924 07:20:23.422871 4045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:23.422937 4045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7623b940634227932d929393238a15, State: Initialized, Role: FOLLOWER
I20260924 07:20:23.423377 4045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7623b940634227932d929393238a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260924 07:20:23.423496 4045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:23.423579 4045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:23.423686 4045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:20:23 dist-test-slave-zx46 krb5kdc[3964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:20:23.424606 4045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7623b940634227932d929393238a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260924 07:20:23.425019 4045 leader_election.cc:302] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f7623b940634227932d929393238a15; no voters:
I20260924 07:20:23.425395 4045 leader_election.cc:288] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:23.425527 4050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:23.425993 4050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [term 1 LEADER]: Becoming Leader. State: Replica: 0f7623b940634227932d929393238a15, State: Running, Role: LEADER
I20260924 07:20:23.426344 4050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7623b940634227932d929393238a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260924 07:20:23.426717 4045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:23.428032 4052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f7623b940634227932d929393238a15. Latest consensus state: current_term: 1 leader_uuid: "0f7623b940634227932d929393238a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7623b940634227932d929393238a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } } }
I20260924 07:20:23.428036 4051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f7623b940634227932d929393238a15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f7623b940634227932d929393238a15" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } } }
I20260924 07:20:23.428206 4052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:23.428206 4051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:23.430568 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:23.415502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40147 (local address 127.0.0.1:40231)
0924 07:20:23.415955 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:23.415969 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:23.416002 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:20:23.417375 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:23.417396 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:23.417787 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:23.418149 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:23.418850 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:23.420320 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:23.421199 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:23.421969 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:23.422148 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:23.424738 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:23.424749 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:23.424765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:23.424794 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:23.426806 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:23.427347 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:23.427353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:23.427358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:23.427506 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:23.427851 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:23.427853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:23.427855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:23.428405 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:23.429267 (+ 862us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:23.429397 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 07:20:23.429613 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":144,"threads_started":1}
W20260924 07:20:23.431768 4065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:23.431836 4065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:23.432315 4061 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:23.433225 4061 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:23.433609 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:23.438990 4061 catalog_manager.cc:1347] Generated new cluster ID: 2323e93f057446b28aa19d8767967bf8
I20260924 07:20:23.439051 4061 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:23.443866 4048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:23.435729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43183 (local address 127.0.0.1:40231)
0924 07:20:23.435830 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:23.435834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:23.436072 (+ 238us) server_negotiation.cc:407] Connection header received
0924 07:20:23.436210 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:23.436218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:23.436255 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:23.436338 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:23.436978 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:23.437695 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:23.438883 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:23.439218 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:23.439332 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:23.440691 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:23.440694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:23.440697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:23.440721 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:23.442218 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:23.442630 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:23.442634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:23.442636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:23.442722 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:23.443001 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:23.443005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:23.443006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:23.443175 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:23.443334 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:23.443570 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 07:20:23.443691 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:20:23.466845 4061 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:23.467907 4061 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:23.474856 4061 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f7623b940634227932d929393238a15: Generated new TSK 0
I20260924 07:20:23.475600 4061 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:23.486773 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 3977
2026-09-24T07:20:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:23 dist-test-slave-zx46 krb5kdc[4076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:23 dist-test-slave-zx46 krb5kdc[4076](info): set up 2 sockets
Sep 24 07:20:23 dist-test-slave-zx46 krb5kdc[4076](info): commencing operation
krb5kdc: starting...
W20260924 07:20:24.529223 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:24 dist-test-slave-zx46 krb5kdc[4076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:24.648216 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:24.756064 4089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:24.756331 4089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:24.756374 4089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:24.760270 4089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:24.760329 4089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:24.760347 4089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:24.760363 4089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:24.760380 4089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:24.765471 4089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:24.766422 4089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:24.767382 4089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:24.775975 4095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:24.775985 4094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:24.775985 4097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:24.776053 4089 server_base.cc:1034] running on GCE node
I20260924 07:20:24.776789 4089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:24.777896 4089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:24.779078 4089 hybrid_clock.cc:648] HybridClock initialized: now 1790234424779050 us; error 51 us; skew 500 ppm
Sep 24 07:20:24 dist-test-slave-zx46 krb5kdc[4076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:24.781683 4089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:24.782546 4089 webserver.cc:460] Webserver started at http://127.0.0.1:42539/ using document root <none> and password file <none>
I20260924 07:20:24.783125 4089 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:24.783216 4089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:24.783450 4089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:24.785581 4089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "e272b52e85314e6c892cee069877b70a"
format_stamp: "Formatted at 2026-09-24 07:20:24 on dist-test-slave-zx46"
server_key: "e1acb7b5af76a6cc8acd9458682403d5"
server_key_iv: "37512d51db693bc5c89bf900aec05da2"
server_key_version: "encryptionkey@0"
I20260924 07:20:24.786154 4089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "e272b52e85314e6c892cee069877b70a"
format_stamp: "Formatted at 2026-09-24 07:20:24 on dist-test-slave-zx46"
server_key: "e1acb7b5af76a6cc8acd9458682403d5"
server_key_iv: "37512d51db693bc5c89bf900aec05da2"
server_key_version: "encryptionkey@0"
I20260924 07:20:24.789922 4089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:20:24.792241 4104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:24.793429 4089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:24.793565 4089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "e272b52e85314e6c892cee069877b70a"
format_stamp: "Formatted at 2026-09-24 07:20:24 on dist-test-slave-zx46"
server_key: "e1acb7b5af76a6cc8acd9458682403d5"
server_key_iv: "37512d51db693bc5c89bf900aec05da2"
server_key_version: "encryptionkey@0"
I20260924 07:20:24.793679 4089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:24.802738 4089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:24.805474 4089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:24.805673 4089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:24.812559 4089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45537
I20260924 07:20:24.812582 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45537 every 8 connection(s)
I20260924 07:20:24.813542 4089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:24.814200 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4089
I20260924 07:20:24.814352 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:24.814638 15870 external_mini_cluster.cc:1428] Setting key cb869d9f855c8ce6a0e7be72420e29ff
I20260924 07:20:24.816737 4157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:24.822254 4157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a: Bootstrap starting.
I20260924 07:20:24.823519 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:24.816022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33807 (local address 127.0.0.1:45537)
0924 07:20:24.816470 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:24.816486 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:24.816521 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:24.817288 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:24.817307 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:24.817643 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:24.817982 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:24.818381 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:24.819346 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:24.820276 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:24.820753 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:24.820922 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:24.821871 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:24.821876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:24.821888 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:24.822263 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:24.822343 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:24.822378 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:24.822492 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:24.822640 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:20:24.822849 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":143,"threads_started":1}
I20260924 07:20:24.824568 4157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:24.825457 4157 log.cc:824] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:24.826478 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:24.827469 4157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a: No bootstrap required, opened a new log
I20260924 07:20:24.832020 4157 raft_consensus.cc:348] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272b52e85314e6c892cee069877b70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45537 } }
I20260924 07:20:24.832340 4157 raft_consensus.cc:374] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:24.832458 4157 raft_consensus.cc:729] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e272b52e85314e6c892cee069877b70a, State: Initialized, Role: FOLLOWER
I20260924 07:20:24.832671 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:24.828480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36795 (local address 127.0.0.1:45537)
0924 07:20:24.828608 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:24.828613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:24.828798 (+ 185us) server_negotiation.cc:407] Connection header received
0924 07:20:24.828908 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:24.828910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:24.828947 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:24.829065 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:24.829323 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:24.829807 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:24.831029 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:24.831268 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:24.831341 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:24.832155 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:24.832158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:24.832160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:24.832170 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:24.832192 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:24.832202 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:24.832260 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:24.832444 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 07:20:24.832504 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:20:24.832980 4157 consensus_queue.cc:260] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272b52e85314e6c892cee069877b70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45537 } }
I20260924 07:20:24.833184 4157 raft_consensus.cc:388] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:24.833253 4157 raft_consensus.cc:482] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:24.833349 4157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:24.834424 4157 raft_consensus.cc:504] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272b52e85314e6c892cee069877b70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45537 } }
I20260924 07:20:24.834728 4157 leader_election.cc:302] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e272b52e85314e6c892cee069877b70a; no voters:
I20260924 07:20:24.834975 4157 leader_election.cc:288] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:24.835083 4167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:24.835346 4167 raft_consensus.cc:686] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [term 1 LEADER]: Becoming Leader. State: Replica: e272b52e85314e6c892cee069877b70a, State: Running, Role: LEADER
I20260924 07:20:24.835742 4167 consensus_queue.cc:237] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272b52e85314e6c892cee069877b70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45537 } }
I20260924 07:20:24.836253 4157 sys_catalog.cc:564] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:24.837517 4169 sys_catalog.cc:455] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e272b52e85314e6c892cee069877b70a. Latest consensus state: current_term: 1 leader_uuid: "e272b52e85314e6c892cee069877b70a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272b52e85314e6c892cee069877b70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45537 } } }
I20260924 07:20:24.837612 4169 sys_catalog.cc:458] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:24.837949 4168 sys_catalog.cc:455] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e272b52e85314e6c892cee069877b70a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272b52e85314e6c892cee069877b70a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45537 } } }
I20260924 07:20:24.838017 4176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:24.838059 4168 sys_catalog.cc:458] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:24.840942 4176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:24.846933 4176 catalog_manager.cc:1347] Generated new cluster ID: 4c37eabdb8854cae814ad830230e6586
I20260924 07:20:24.847007 4176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:24.861768 4176 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:24.862788 4176 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:24.872404 4176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e272b52e85314e6c892cee069877b70a: Generated new TSK 0
I20260924 07:20:24.873178 4176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:24.883441 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4089
2026-09-24T07:20:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:24 dist-test-slave-zx46 krb5kdc[4188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:24 dist-test-slave-zx46 krb5kdc[4188](info): set up 2 sockets
Sep 24 07:20:24 dist-test-slave-zx46 krb5kdc[4188](info): commencing operation
krb5kdc: starting...
W20260924 07:20:25.921247 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:26 dist-test-slave-zx46 krb5kdc[4188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:26.049788 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35969
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:26.158082 4201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:26.158347 4201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:26.158389 4201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:26.162312 4201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:26.162362 4201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:26.162381 4201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:26.162397 4201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:26.162412 4201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:26.167483 4201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:26.168632 4201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:26.169617 4201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:26.178056 4206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:26.178042 4207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:26.178042 4209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:26.178185 4201 server_base.cc:1034] running on GCE node
I20260924 07:20:26.178938 4201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:26.180023 4201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:26.181185 4201 hybrid_clock.cc:648] HybridClock initialized: now 1790234426181170 us; error 24 us; skew 500 ppm
Sep 24 07:20:26 dist-test-slave-zx46 krb5kdc[4188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:26.183820 4201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:26.184646 4201 webserver.cc:460] Webserver started at http://127.0.0.1:41007/ using document root <none> and password file <none>
I20260924 07:20:26.185266 4201 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:26.185356 4201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:26.185581 4201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:26.187633 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "98a5cddc604d48dfb3dd327f604cfc46"
format_stamp: "Formatted at 2026-09-24 07:20:26 on dist-test-slave-zx46"
server_key: "d41c5a00750022c9bb8e470879b3feea"
server_key_iv: "b9c426e790d71c979aae2060399bad81"
server_key_version: "encryptionkey@0"
I20260924 07:20:26.188199 4201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "98a5cddc604d48dfb3dd327f604cfc46"
format_stamp: "Formatted at 2026-09-24 07:20:26 on dist-test-slave-zx46"
server_key: "d41c5a00750022c9bb8e470879b3feea"
server_key_iv: "b9c426e790d71c979aae2060399bad81"
server_key_version: "encryptionkey@0"
I20260924 07:20:26.191843 4201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 07:20:26.194221 4216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:26.195356 4201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:26.195492 4201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "98a5cddc604d48dfb3dd327f604cfc46"
format_stamp: "Formatted at 2026-09-24 07:20:26 on dist-test-slave-zx46"
server_key: "d41c5a00750022c9bb8e470879b3feea"
server_key_iv: "b9c426e790d71c979aae2060399bad81"
server_key_version: "encryptionkey@0"
I20260924 07:20:26.195605 4201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:26.200030 4201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:26.202805 4201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:26.202993 4201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:26.209487 4201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35969
I20260924 07:20:26.209509 4268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35969 every 8 connection(s)
I20260924 07:20:26.210355 4201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:26.212895 4269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:26.215444 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4201
I20260924 07:20:26.215603 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:26.215886 15870 external_mini_cluster.cc:1428] Setting key fe36702a5f2a08e391a46d225399d4c0
I20260924 07:20:26.218984 4269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46: Bootstrap starting.
I20260924 07:20:26.220849 4272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:26.217299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40109 (local address 127.0.0.1:35969)
0924 07:20:26.217715 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:26.217728 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:26.217762 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:26.218401 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:26.218436 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:26.218705 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:26.219035 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:26.219772 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:26.219776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:26.219787 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:26.219797 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:26.219845 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:26.219863 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:26.220004 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:26.220069 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:20:26.220220 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":137,"threads_started":1}
I20260924 07:20:26.221400 4269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:26.222301 4269 log.cc:824] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:26.223640 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:26.224417 4269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46: No bootstrap required, opened a new log
I20260924 07:20:26.227331 4272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:26.225792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51365 (local address 127.0.0.1:35969)
0924 07:20:26.225918 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:26.225922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:26.226065 (+ 143us) server_negotiation.cc:407] Connection header received
0924 07:20:26.226171 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:26.226174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:26.226209 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:26.226285 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:26.226966 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:26.226970 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:26.226972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:26.226984 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:26.227011 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:26.227023 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:26.227115 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:26.227178 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:20:26.227215 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:20:26.228611 4269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5cddc604d48dfb3dd327f604cfc46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260924 07:20:26.228812 4269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:26.228844 4269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a5cddc604d48dfb3dd327f604cfc46, State: Initialized, Role: FOLLOWER
I20260924 07:20:26.229249 4269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5cddc604d48dfb3dd327f604cfc46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260924 07:20:26.229365 4269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:26.229409 4269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:26.229475 4269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:26.230281 4269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5cddc604d48dfb3dd327f604cfc46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260924 07:20:26.230559 4269 leader_election.cc:302] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98a5cddc604d48dfb3dd327f604cfc46; no voters:
I20260924 07:20:26.230785 4269 leader_election.cc:288] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:26.230901 4279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:26.231105 4279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [term 1 LEADER]: Becoming Leader. State: Replica: 98a5cddc604d48dfb3dd327f604cfc46, State: Running, Role: LEADER
I20260924 07:20:26.231477 4279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5cddc604d48dfb3dd327f604cfc46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } }
I20260924 07:20:26.231977 4269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:26.232967 4280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98a5cddc604d48dfb3dd327f604cfc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5cddc604d48dfb3dd327f604cfc46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } } }
I20260924 07:20:26.233117 4280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:26.233503 4281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98a5cddc604d48dfb3dd327f604cfc46. Latest consensus state: current_term: 1 leader_uuid: "98a5cddc604d48dfb3dd327f604cfc46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98a5cddc604d48dfb3dd327f604cfc46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35969 } } }
I20260924 07:20:26.233654 4281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:26.234357 4288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:26.237296 4288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:26.242053 4288 catalog_manager.cc:1347] Generated new cluster ID: 54926d2957b74e1b91ca8bdbe9f2fa58
I20260924 07:20:26.242115 4288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:26.255853 4288 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:26.256193 4288 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:26.266304 4288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98a5cddc604d48dfb3dd327f604cfc46: Generated new TSK 0
I20260924 07:20:26.267171 4288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:26.272080 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4201
2026-09-24T07:20:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:26 dist-test-slave-zx46 krb5kdc[4300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:26 dist-test-slave-zx46 krb5kdc[4300](info): set up 2 sockets
Sep 24 07:20:26 dist-test-slave-zx46 krb5kdc[4300](info): commencing operation
krb5kdc: starting...
W20260924 07:20:27.297160 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:27 dist-test-slave-zx46 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:27.415887 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:27.523598 4313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:27.524046 4313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:27.524140 4313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:27.528388 4313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:27.528453 4313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:27.528473 4313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:27.528489 4313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:27.528504 4313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:27.533774 4313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:27.534744 4313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:27.535702 4313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:27.544086 4321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:27.544124 4319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:27.544387 4313 server_base.cc:1034] running on GCE node
W20260924 07:20:27.544124 4318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:27.544960 4313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:27.546016 4313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:27.547204 4313 hybrid_clock.cc:648] HybridClock initialized: now 1790234427547173 us; error 44 us; skew 500 ppm
Sep 24 07:20:27 dist-test-slave-zx46 krb5kdc[4300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234427, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:27.549783 4313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:27.550695 4313 webserver.cc:460] Webserver started at http://127.0.0.1:41433/ using document root <none> and password file <none>
I20260924 07:20:27.551256 4313 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:27.551343 4313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:27.551585 4313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:27.553686 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "ffc98e9070df4f4ba0e4d3cec73f0772"
format_stamp: "Formatted at 2026-09-24 07:20:27 on dist-test-slave-zx46"
server_key: "e32595cd968668d3f5e4b6a3f15f0c7c"
server_key_iv: "691cb493c356f00fbb9c482ff1029d1f"
server_key_version: "encryptionkey@0"
I20260924 07:20:27.554224 4313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "ffc98e9070df4f4ba0e4d3cec73f0772"
format_stamp: "Formatted at 2026-09-24 07:20:27 on dist-test-slave-zx46"
server_key: "e32595cd968668d3f5e4b6a3f15f0c7c"
server_key_iv: "691cb493c356f00fbb9c482ff1029d1f"
server_key_version: "encryptionkey@0"
I20260924 07:20:27.558058 4313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 07:20:27.562203 4328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:27.564210 4313 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 07:20:27.564352 4313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "ffc98e9070df4f4ba0e4d3cec73f0772"
format_stamp: "Formatted at 2026-09-24 07:20:27 on dist-test-slave-zx46"
server_key: "e32595cd968668d3f5e4b6a3f15f0c7c"
server_key_iv: "691cb493c356f00fbb9c482ff1029d1f"
server_key_version: "encryptionkey@0"
I20260924 07:20:27.564482 4313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:27.597785 4313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:27.602212 4313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:27.602424 4313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:27.610932 4313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33829
I20260924 07:20:27.610939 4380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33829 every 8 connection(s)
I20260924 07:20:27.612040 4313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:27.612534 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4313
I20260924 07:20:27.612699 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:27.613039 15870 external_mini_cluster.cc:1428] Setting key c90fbfe7bcac42f9dfce9c89db752656
I20260924 07:20:27.615494 4381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 07:20:27 dist-test-slave-zx46 krb5kdc[4300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:20:27.622836 4381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772: Bootstrap starting.
I20260924 07:20:27.625887 4381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:27.626876 4381 log.cc:824] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:27.628983 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:27.614487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:40897 (local address 127.0.0.1:33829)
0924 07:20:27.614906 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:27.614921 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:27.614957 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:20:27.615672 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:27.615691 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:27.616040 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:27.616315 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:27.617330 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:27.618314 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:27.619592 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:27.620054 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:27.620172 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:27.623116 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:27.623123 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:27.623140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:27.623174 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:27.625391 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:27.626117 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:27.626124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:27.626132 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:27.626285 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:27.626607 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:27.626610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:27.626611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:27.627196 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:27.627419 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:27.627773 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 07:20:27.628023 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":150,"threads_started":1}
I20260924 07:20:27.629276 4381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772: No bootstrap required, opened a new log
I20260924 07:20:27.631987 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:27.633764 4381 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260924 07:20:27.634078 4381 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:27.634137 4381 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc98e9070df4f4ba0e4d3cec73f0772, State: Initialized, Role: FOLLOWER
I20260924 07:20:27.634727 4381 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260924 07:20:27.634923 4381 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:27.635001 4381 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:27.635129 4381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:27.636204 4381 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260924 07:20:27.636598 4381 leader_election.cc:302] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc98e9070df4f4ba0e4d3cec73f0772; no voters:
I20260924 07:20:27.636915 4381 leader_election.cc:288] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:27.637073 4391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:27.637341 4391 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [term 1 LEADER]: Becoming Leader. State: Replica: ffc98e9070df4f4ba0e4d3cec73f0772, State: Running, Role: LEADER
I20260924 07:20:27.637693 4391 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } }
I20260924 07:20:27.638504 4381 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:27.638911 4391 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc98e9070df4f4ba0e4d3cec73f0772. Latest consensus state: current_term: 1 leader_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260924 07:20:27.639020 4391 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:27.640753 4393 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc98e9070df4f4ba0e4d3cec73f0772" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33829 } } }
I20260924 07:20:27.640851 4393 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:27.643576 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:27.634005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:60515 (local address 127.0.0.1:33829)
0924 07:20:27.634121 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:27.634124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:27.634365 (+ 241us) server_negotiation.cc:407] Connection header received
0924 07:20:27.634503 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:27.634505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:27.634541 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:27.634601 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:27.635478 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:27.635976 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:27.637258 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:27.637610 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:27.637722 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:27.639533 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:27.639541 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:27.639546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:27.639578 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:27.641599 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:27.642164 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:27.642167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:27.642168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:27.642238 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:27.642638 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:27.642642 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:27.642643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:27.642841 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:27.642987 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:27.643314 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 07:20:27.643421 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 07:20:27.644224 4407 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:27.644316 4407 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:27.644413 4395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:27.645287 4395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:27.650693 4395 catalog_manager.cc:1347] Generated new cluster ID: 47fc3b759f834808ba5f8641deb72ae3
I20260924 07:20:27.650753 4395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:27.662262 4395 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:27.663200 4395 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:27.670537 4395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc98e9070df4f4ba0e4d3cec73f0772: Generated new TSK 0
I20260924 07:20:27.671202 4395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:27.681430 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4313
W20260924 07:20:27.687041 4298 connection.cc:535] client connection to 127.0.0.1:33829 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33829): Connection reset by peer (error 104)
2026-09-24T07:20:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:27 dist-test-slave-zx46 krb5kdc[4412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:27 dist-test-slave-zx46 krb5kdc[4412](info): set up 2 sockets
Sep 24 07:20:27 dist-test-slave-zx46 krb5kdc[4412](info): commencing operation
krb5kdc: starting...
W20260924 07:20:28.701140 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:28 dist-test-slave-zx46 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:28.815945 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:28.923823 4425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:28.924085 4425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:28.924151 4425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:28.928115 4425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:28.928175 4425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:28.928195 4425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:28.928211 4425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:28.928226 4425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:28.933379 4425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:28.934365 4425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:28.935329 4425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:28.943370 4433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:28.943372 4430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:28.943534 4431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:28.943606 4425 server_base.cc:1034] running on GCE node
I20260924 07:20:28.944321 4425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:28.945472 4425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:28.946623 4425 hybrid_clock.cc:648] HybridClock initialized: now 1790234428946598 us; error 40 us; skew 500 ppm
Sep 24 07:20:28 dist-test-slave-zx46 krb5kdc[4412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:28.953150 4425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:28.954072 4425 webserver.cc:460] Webserver started at http://127.0.0.1:40429/ using document root <none> and password file <none>
I20260924 07:20:28.954627 4425 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:28.954685 4425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:28.954905 4425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:28.956948 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "3e36d7cbe6aa491ab9a406d6354dfa78"
format_stamp: "Formatted at 2026-09-24 07:20:28 on dist-test-slave-zx46"
server_key: "988ee65fabd464fc59031c616838e36c"
server_key_iv: "ceee456957429e64335f7eedd59882f7"
server_key_version: "encryptionkey@0"
I20260924 07:20:28.957572 4425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "3e36d7cbe6aa491ab9a406d6354dfa78"
format_stamp: "Formatted at 2026-09-24 07:20:28 on dist-test-slave-zx46"
server_key: "988ee65fabd464fc59031c616838e36c"
server_key_iv: "ceee456957429e64335f7eedd59882f7"
server_key_version: "encryptionkey@0"
I20260924 07:20:28.961199 4425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:28.963493 4440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:28.964664 4425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:28.964783 4425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "3e36d7cbe6aa491ab9a406d6354dfa78"
format_stamp: "Formatted at 2026-09-24 07:20:28 on dist-test-slave-zx46"
server_key: "988ee65fabd464fc59031c616838e36c"
server_key_iv: "ceee456957429e64335f7eedd59882f7"
server_key_version: "encryptionkey@0"
I20260924 07:20:28.964876 4425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:28.987354 4425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:28.990242 4425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:28.990458 4425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:28.997630 4425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40751
I20260924 07:20:28.997648 4492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40751 every 8 connection(s)
I20260924 07:20:28.998564 4425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:29.001286 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:29.001315 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4425
I20260924 07:20:29.001444 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:29.001727 15870 external_mini_cluster.cc:1428] Setting key b2a4cc7581fe4ed67329364b4212c946
I20260924 07:20:29.008242 4493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78: Bootstrap starting.
I20260924 07:20:29.010890 4493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:29.011737 4493 log.cc:824] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:29.011894 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:29.003130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:51747 (local address 127.0.0.1:40751)
0924 07:20:29.003548 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:29.003567 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:29.003621 (+ 54us) server_negotiation.cc:407] Connection header received
0924 07:20:29.004599 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:29.004623 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:29.005108 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:29.005472 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:29.005876 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:29.007318 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:29.008220 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:29.008892 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:29.009071 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:29.009914 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:29.009920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:29.009935 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:29.010406 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:29.010469 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:29.010498 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:29.010653 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:29.010771 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 07:20:29.010955 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260924 07:20:29.013677 4493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78: No bootstrap required, opened a new log
I20260924 07:20:29.014829 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:29.017511 4493 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 07:20:29.017745 4493 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:29.017817 4493 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e36d7cbe6aa491ab9a406d6354dfa78, State: Initialized, Role: FOLLOWER
I20260924 07:20:29.018234 4493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 07:20:29.018370 4493 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:29.018472 4493 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:29.018586 4493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:29.019619 4493 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 07:20:29.020006 4493 leader_election.cc:302] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e36d7cbe6aa491ab9a406d6354dfa78; no voters:
I20260924 07:20:29.020366 4493 leader_election.cc:288] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:29.020489 4503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:29.020745 4503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [term 1 LEADER]: Becoming Leader. State: Replica: 3e36d7cbe6aa491ab9a406d6354dfa78, State: Running, Role: LEADER
I20260924 07:20:29.021299 4496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:29.016786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:36677 (local address 127.0.0.1:40751)
0924 07:20:29.016907 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:29.016911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:29.017155 (+ 244us) server_negotiation.cc:407] Connection header received
0924 07:20:29.017271 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:29.017274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:29.017318 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:29.017396 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:29.017660 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:29.018321 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:29.019409 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:29.019648 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:29.019726 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:29.020583 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:29.020587 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:29.020590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:29.020604 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:29.020632 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:29.020649 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:29.020744 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:29.021140 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 07:20:29.021177 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:20:29.021284 4503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } }
I20260924 07:20:29.021636 4493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:29.022902 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260924 07:20:29.023031 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:29.023407 4514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:29.023384 4505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e36d7cbe6aa491ab9a406d6354dfa78. Latest consensus state: current_term: 1 leader_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e36d7cbe6aa491ab9a406d6354dfa78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40751 } } }
I20260924 07:20:29.024204 4505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:29.026360 4514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:29.031719 4514 catalog_manager.cc:1347] Generated new cluster ID: 12a67a7ac93a4915bb9698b51ba0dfcb
I20260924 07:20:29.031787 4514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:29.038529 4514 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:29.039371 4514 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:29.045255 4514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e36d7cbe6aa491ab9a406d6354dfa78: Generated new TSK 0
I20260924 07:20:29.046038 4514 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:29.055171 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4425
W20260924 07:20:29.060513 4410 connection.cc:535] client connection to 127.0.0.1:40751 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40751): Connection reset by peer (error 104)
2026-09-24T07:20:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:29 dist-test-slave-zx46 krb5kdc[4524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:29 dist-test-slave-zx46 krb5kdc[4524](info): set up 2 sockets
Sep 24 07:20:29 dist-test-slave-zx46 krb5kdc[4524](info): commencing operation
krb5kdc: starting...
W20260924 07:20:30.065404 15870 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:30 dist-test-slave-zx46 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:30.183450 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:30.289819 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:30.290055 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:30.290093 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:30.294059 4537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:30.294113 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:30.294131 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:30.294176 4537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:30.294209 4537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:30.299420 4537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:30.300421 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:30.301438 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:30.309371 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:30.309376 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:30.309459 4537 server_base.cc:1034] running on GCE node
W20260924 07:20:30.309387 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:30.310158 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:30.311149 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:30.312341 4537 hybrid_clock.cc:648] HybridClock initialized: now 1790234430312330 us; error 32 us; skew 500 ppm
Sep 24 07:20:30 dist-test-slave-zx46 krb5kdc[4524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:30.314941 4537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:30.315778 4537 webserver.cc:460] Webserver started at http://127.0.0.1:44371/ using document root <none> and password file <none>
I20260924 07:20:30.316320 4537 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:30.316416 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:30.316635 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:30.318624 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "a7e9f05e93494416ba71bbb02d2b49c7"
format_stamp: "Formatted at 2026-09-24 07:20:30 on dist-test-slave-zx46"
server_key: "415118656d194bfc7631ec30f61ef08e"
server_key_iv: "a0d09493a600e4f5ba993c77d05a7c61"
server_key_version: "encryptionkey@0"
I20260924 07:20:30.319165 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "a7e9f05e93494416ba71bbb02d2b49c7"
format_stamp: "Formatted at 2026-09-24 07:20:30 on dist-test-slave-zx46"
server_key: "415118656d194bfc7631ec30f61ef08e"
server_key_iv: "a0d09493a600e4f5ba993c77d05a7c61"
server_key_version: "encryptionkey@0"
I20260924 07:20:30.322741 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260924 07:20:30.325076 4552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:30.326189 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:20:30.326318 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "a7e9f05e93494416ba71bbb02d2b49c7"
format_stamp: "Formatted at 2026-09-24 07:20:30 on dist-test-slave-zx46"
server_key: "415118656d194bfc7631ec30f61ef08e"
server_key_iv: "a0d09493a600e4f5ba993c77d05a7c61"
server_key_version: "encryptionkey@0"
I20260924 07:20:30.326416 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:30.330822 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:30.333596 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:30.333768 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:30.340629 4537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46265
I20260924 07:20:30.340647 4604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46265 every 8 connection(s)
I20260924 07:20:30.341575 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:30.344138 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:30.349072 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4537
I20260924 07:20:30.349215 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:30.349481 15870 external_mini_cluster.cc:1428] Setting key 6b7b324f473361d65c1bc61adc34daa4
I20260924 07:20:30.349695 4605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7: Bootstrap starting.
I20260924 07:20:30.352088 4605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:30.352983 4605 log.cc:824] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:30.354241 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:30.350664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:52511 (local address 127.0.0.1:46265)
0924 07:20:30.351105 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:30.351119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:30.351151 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:20:30.351727 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:30.351743 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:30.351995 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:30.352322 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:30.353100 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:30.353105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:30.353116 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:30.353126 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:30.353202 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:30.353230 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:30.353338 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:30.353440 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:20:30.353602 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":150,"threads_started":1}
I20260924 07:20:30.354969 4605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7: No bootstrap required, opened a new log
I20260924 07:20:30.356869 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:30.359130 4605 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260924 07:20:30.359449 4605 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:30.359515 4605 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e9f05e93494416ba71bbb02d2b49c7, State: Initialized, Role: FOLLOWER
I20260924 07:20:30.360001 4605 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260924 07:20:30.360172 4605 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:30.360253 4605 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:30.360369 4605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:30.360523 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:30.358875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:45573 (local address 127.0.0.1:46265)
0924 07:20:30.359004 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:30.359008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:30.359076 (+ 68us) server_negotiation.cc:407] Connection header received
0924 07:20:30.359221 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:30.359225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:30.359260 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:30.359358 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:30.360040 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:30.360044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:30.360047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:30.360060 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:30.360089 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:30.360103 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:30.360201 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:30.360343 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:20:30.360390 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:20:30.361589 4605 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260924 07:20:30.361912 4605 leader_election.cc:302] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7e9f05e93494416ba71bbb02d2b49c7; no voters:
I20260924 07:20:30.362156 4605 leader_election.cc:288] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:30.362273 4615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:30.362491 4615 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [term 1 LEADER]: Becoming Leader. State: Replica: a7e9f05e93494416ba71bbb02d2b49c7, State: Running, Role: LEADER
I20260924 07:20:30.362829 4615 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } }
I20260924 07:20:30.363348 4605 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:30.364378 4616 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260924 07:20:30.364516 4616 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:30.364866 4624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:30.365195 4617 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e9f05e93494416ba71bbb02d2b49c7. Latest consensus state: current_term: 1 leader_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7e9f05e93494416ba71bbb02d2b49c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46265 } } }
I20260924 07:20:30.365409 4617 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:30.367853 4624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:30.372784 4624 catalog_manager.cc:1347] Generated new cluster ID: b65bcf5dd0814860a33b1c0da98a747e
I20260924 07:20:30.372848 4624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:30.392195 4624 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:30.392481 4624 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:30.400511 4624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7e9f05e93494416ba71bbb02d2b49c7: Generated new TSK 0
I20260924 07:20:30.401226 4624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:30.407282 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4537
2026-09-24T07:20:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:30 dist-test-slave-zx46 krb5kdc[4636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:30 dist-test-slave-zx46 krb5kdc[4636](info): set up 2 sockets
Sep 24 07:20:30 dist-test-slave-zx46 krb5kdc[4636](info): commencing operation
krb5kdc: starting...
W20260924 07:20:31.429631 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:31 dist-test-slave-zx46 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:31.551501 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40179
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:31.659514 4649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:31.659765 4649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:31.659807 4649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:31.663715 4649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:31.663769 4649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:31.663791 4649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:31.663810 4649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:31.663827 4649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:31.668918 4649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40179
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:31.669919 4649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:31.670852 4649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:31.679276 4655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:31.679298 4657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:31.679466 4649 server_base.cc:1034] running on GCE node
W20260924 07:20:31.679294 4654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:31.680073 4649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:31.681166 4649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:31.682348 4649 hybrid_clock.cc:648] HybridClock initialized: now 1790234431682337 us; error 27 us; skew 500 ppm
Sep 24 07:20:31 dist-test-slave-zx46 krb5kdc[4636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234431, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:31.685137 4649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:31.685988 4649 webserver.cc:460] Webserver started at http://127.0.0.1:35649/ using document root <none> and password file <none>
I20260924 07:20:31.686537 4649 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:31.686635 4649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:31.686892 4649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:31.688961 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "82e224c7e1b64fdfab59662097119c68"
format_stamp: "Formatted at 2026-09-24 07:20:31 on dist-test-slave-zx46"
server_key: "01c176ae24e770d45ddaa55a923a18f7"
server_key_iv: "70d7b16a2c9ab8b574d69ba8b5a33127"
server_key_version: "encryptionkey@0"
I20260924 07:20:31.689594 4649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "82e224c7e1b64fdfab59662097119c68"
format_stamp: "Formatted at 2026-09-24 07:20:31 on dist-test-slave-zx46"
server_key: "01c176ae24e770d45ddaa55a923a18f7"
server_key_iv: "70d7b16a2c9ab8b574d69ba8b5a33127"
server_key_version: "encryptionkey@0"
I20260924 07:20:31.693164 4649 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:31.695395 4664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:31.696547 4649 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:31.696691 4649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "82e224c7e1b64fdfab59662097119c68"
format_stamp: "Formatted at 2026-09-24 07:20:31 on dist-test-slave-zx46"
server_key: "01c176ae24e770d45ddaa55a923a18f7"
server_key_iv: "70d7b16a2c9ab8b574d69ba8b5a33127"
server_key_version: "encryptionkey@0"
I20260924 07:20:31.696796 4649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:31.719341 4649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:31.722220 4649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:31.722443 4649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:31.729584 4649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40179
I20260924 07:20:31.729600 4716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40179 every 8 connection(s)
I20260924 07:20:31.730527 4649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:31.733300 4717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:31.737936 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4649
I20260924 07:20:31.738088 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:31.738354 15870 external_mini_cluster.cc:1428] Setting key 2beb5c840ecd5afe77f08f70b81032dd
I20260924 07:20:31.738826 4717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68: Bootstrap starting.
I20260924 07:20:31.741355 4717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:31.742072 4717 log.cc:824] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:31.743963 4717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68: No bootstrap required, opened a new log
Sep 24 07:20:31 dist-test-slave-zx46 krb5kdc[4636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:20:31.747957 4717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e224c7e1b64fdfab59662097119c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260924 07:20:31.748178 4717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:31.748265 4717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e224c7e1b64fdfab59662097119c68, State: Initialized, Role: FOLLOWER
I20260924 07:20:31.748879 4717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e224c7e1b64fdfab59662097119c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260924 07:20:31.749100 4717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:31.749181 4717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:31.749306 4717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:31.750515 4717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e224c7e1b64fdfab59662097119c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260924 07:20:31.750890 4717 leader_election.cc:302] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82e224c7e1b64fdfab59662097119c68; no voters:
I20260924 07:20:31.751256 4717 leader_election.cc:288] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:31.751370 4722 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:31.751622 4722 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [term 1 LEADER]: Becoming Leader. State: Replica: 82e224c7e1b64fdfab59662097119c68, State: Running, Role: LEADER
I20260924 07:20:31.751991 4722 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e224c7e1b64fdfab59662097119c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } }
I20260924 07:20:31.752609 4717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:31.753526 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:31.739723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:33901 (local address 127.0.0.1:40179)
0924 07:20:31.740132 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:31.740147 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:31.740182 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:20:31.740830 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:31.740845 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:31.741260 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:31.741606 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:31.742431 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:31.743412 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:31.744325 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:31.744813 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:31.744990 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:31.747833 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:31.747838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:31.747851 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:31.747877 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:31.750000 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:31.750515 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:31.750522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:31.750531 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:31.750673 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:31.750999 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:31.751004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:31.751007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:31.751536 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:31.751741 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:31.752237 (+ 496us) server_negotiation.cc:299] Negotiation successful
0924 07:20:31.752484 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":124,"threads_started":1}
I20260924 07:20:31.754318 4723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82e224c7e1b64fdfab59662097119c68. Latest consensus state: current_term: 1 leader_uuid: "82e224c7e1b64fdfab59662097119c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e224c7e1b64fdfab59662097119c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } } }
I20260924 07:20:31.754434 4723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:31.754776 4724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82e224c7e1b64fdfab59662097119c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82e224c7e1b64fdfab59662097119c68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40179 } } }
I20260924 07:20:31.755097 4724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:31.757262 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 07:20:31.757613 4737 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:31.757668 4737 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:31.757753 4736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:31.758598 4736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:31.764909 4736 catalog_manager.cc:1347] Generated new cluster ID: c750e5c2987e44228ad374f3ec577d6c
I20260924 07:20:31.765045 4736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:31.768603 4720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:31.759517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:58839 (local address 127.0.0.1:40179)
0924 07:20:31.759628 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:31.759633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:31.759870 (+ 237us) server_negotiation.cc:407] Connection header received
0924 07:20:31.759977 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:31.759981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:31.760030 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:31.760125 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:31.760961 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:31.761765 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:31.762864 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:31.763235 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:31.763342 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:31.764792 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:31.764795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:31.764798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:31.764824 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:31.766733 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:31.767311 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:31.767314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:31.767316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:31.767393 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:31.767748 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:31.767751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:31.767753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:31.767936 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:31.768056 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:31.768362 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 07:20:31.768452 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:20:31.787886 4736 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:31.788808 4736 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:31.797863 4736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 82e224c7e1b64fdfab59662097119c68: Generated new TSK 0
I20260924 07:20:31.798573 4736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:31.805998 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4649
W20260924 07:20:31.811553 4634 connection.cc:535] client connection to 127.0.0.1:40179 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40179): Connection reset by peer (error 104)
2026-09-24T07:20:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:31 dist-test-slave-zx46 krb5kdc[4748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:31 dist-test-slave-zx46 krb5kdc[4748](info): set up 2 sockets
Sep 24 07:20:31 dist-test-slave-zx46 krb5kdc[4748](info): commencing operation
krb5kdc: starting...
W20260924 07:20:32.829331 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:32 dist-test-slave-zx46 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:32.948891 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45519
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:33.055358 4761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:33.055645 4761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:33.055718 4761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:33.059566 4761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:33.059659 4761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:33.059696 4761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:33.059729 4761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:33.059762 4761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:33.064834 4761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45519
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:33.065896 4761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:33.066917 4761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:33.075476 4767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:33.075528 4761 server_base.cc:1034] running on GCE node
W20260924 07:20:33.075482 4769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:33.075482 4766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:33.076300 4761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:33.077431 4761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:33.078584 4761 hybrid_clock.cc:648] HybridClock initialized: now 1790234433078552 us; error 44 us; skew 500 ppm
Sep 24 07:20:33 dist-test-slave-zx46 krb5kdc[4748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:33.081274 4761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:33.082161 4761 webserver.cc:460] Webserver started at http://127.0.0.1:39691/ using document root <none> and password file <none>
I20260924 07:20:33.082706 4761 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:33.082799 4761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:33.083029 4761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:33.085098 4761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "b140329da5cb4fc1aaa2bf9e3bf55316"
format_stamp: "Formatted at 2026-09-24 07:20:33 on dist-test-slave-zx46"
server_key: "a10d9e4255b81bcca741aef584a2b6ff"
server_key_iv: "b0339071cb9dd30349174bb4e380f961"
server_key_version: "encryptionkey@0"
I20260924 07:20:33.085666 4761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "b140329da5cb4fc1aaa2bf9e3bf55316"
format_stamp: "Formatted at 2026-09-24 07:20:33 on dist-test-slave-zx46"
server_key: "a10d9e4255b81bcca741aef584a2b6ff"
server_key_iv: "b0339071cb9dd30349174bb4e380f961"
server_key_version: "encryptionkey@0"
I20260924 07:20:33.089226 4761 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:33.091642 4776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:33.092814 4761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:33.092953 4761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "b140329da5cb4fc1aaa2bf9e3bf55316"
format_stamp: "Formatted at 2026-09-24 07:20:33 on dist-test-slave-zx46"
server_key: "a10d9e4255b81bcca741aef584a2b6ff"
server_key_iv: "b0339071cb9dd30349174bb4e380f961"
server_key_version: "encryptionkey@0"
I20260924 07:20:33.093116 4761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:33.123494 4761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:33.126490 4761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:33.126715 4761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:33.133852 4761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45519
I20260924 07:20:33.133872 4828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45519 every 8 connection(s)
I20260924 07:20:33.134778 4761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:33.134997 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4761
I20260924 07:20:33.135128 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:33.135471 15870 external_mini_cluster.cc:1428] Setting key 8b27b4687f9231e68d6b84dfae889cd5
I20260924 07:20:33.138092 4829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:33.144665 4829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316: Bootstrap starting.
I20260924 07:20:33.145373 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:33.136854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:36031 (local address 127.0.0.1:45519)
0924 07:20:33.137327 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:33.137345 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:33.137386 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:20:33.138227 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:33.138249 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:33.138651 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:33.138992 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:33.139390 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:33.140846 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:33.141803 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:33.142469 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:33.142629 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:33.143472 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:33.143478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:33.143495 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:33.144000 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:33.144074 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:33.144102 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:33.144228 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:33.144352 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 07:20:33.144473 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":188,"threads_started":1}
I20260924 07:20:33.147172 4829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:33.147948 4829 log.cc:824] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:33.148275 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:33.149919 4829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316: No bootstrap required, opened a new log
I20260924 07:20:33.154615 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:33.150388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:46833 (local address 127.0.0.1:45519)
0924 07:20:33.150488 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:33.150492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:33.150745 (+ 253us) server_negotiation.cc:407] Connection header received
0924 07:20:33.150859 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:33.150862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:33.150904 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:33.150988 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:33.151214 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:33.151989 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:33.152956 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:33.153314 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:33.153423 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:33.154208 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:33.154212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:33.154215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:33.154227 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:33.154255 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:33.154267 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:33.154355 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:33.154457 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:20:33.154494 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:20:33.154443 4829 raft_consensus.cc:348] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20260924 07:20:33.154723 4829 raft_consensus.cc:374] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:33.154778 4829 raft_consensus.cc:729] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b140329da5cb4fc1aaa2bf9e3bf55316, State: Initialized, Role: FOLLOWER
I20260924 07:20:33.155225 4829 consensus_queue.cc:260] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20260924 07:20:33.155352 4829 raft_consensus.cc:388] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:33.155440 4829 raft_consensus.cc:482] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:33.155571 4829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:33.156481 4829 raft_consensus.cc:504] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20260924 07:20:33.156829 4829 leader_election.cc:302] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b140329da5cb4fc1aaa2bf9e3bf55316; no voters:
I20260924 07:20:33.157189 4829 leader_election.cc:288] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:33.157311 4839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:33.157578 4839 raft_consensus.cc:686] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [term 1 LEADER]: Becoming Leader. State: Replica: b140329da5cb4fc1aaa2bf9e3bf55316, State: Running, Role: LEADER
I20260924 07:20:33.157931 4839 consensus_queue.cc:237] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } }
I20260924 07:20:33.158417 4829 sys_catalog.cc:564] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:33.159559 4840 sys_catalog.cc:455] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } } }
I20260924 07:20:33.159694 4840 sys_catalog.cc:458] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:33.160050 4841 sys_catalog.cc:455] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b140329da5cb4fc1aaa2bf9e3bf55316. Latest consensus state: current_term: 1 leader_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b140329da5cb4fc1aaa2bf9e3bf55316" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45519 } } }
I20260924 07:20:33.160197 4841 sys_catalog.cc:458] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316 [sys.catalog]: This master's current role is: LEADER
W20260924 07:20:33.163051 4854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:33.163120 4854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:33.163203 4848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:33.163889 4848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:33.169291 4848 catalog_manager.cc:1347] Generated new cluster ID: 16162096249a459b94159d9ba3cc6c7e
I20260924 07:20:33.169350 4848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:33.193044 4848 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:33.194272 4848 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:33.206125 4848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b140329da5cb4fc1aaa2bf9e3bf55316: Generated new TSK 0
I20260924 07:20:33.206838 4848 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:33.212886 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4761
W20260924 07:20:33.218171 4746 connection.cc:535] client connection to 127.0.0.1:45519 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45519): Connection reset by peer (error 104)
2026-09-24T07:20:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:33 dist-test-slave-zx46 krb5kdc[4860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:33 dist-test-slave-zx46 krb5kdc[4860](info): set up 2 sockets
Sep 24 07:20:33 dist-test-slave-zx46 krb5kdc[4860](info): commencing operation
krb5kdc: starting...
W20260924 07:20:34.241099 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:34 dist-test-slave-zx46 krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:20:34.360526 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:34.465431 4873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:34.465665 4873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:34.465705 4873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:34.469640 4873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:34.469691 4873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:34.469710 4873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:34.469739 4873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:34.469754 4873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:34.475350 4873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:34.476284 4873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:34.477434 4873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:34.485955 4881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:34.485973 4879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:34.485976 4878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:34.486233 4873 server_base.cc:1034] running on GCE node
I20260924 07:20:34.486886 4873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:34.487905 4873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:34.489089 4873 hybrid_clock.cc:648] HybridClock initialized: now 1790234434489066 us; error 28 us; skew 500 ppm
Sep 24 07:20:34 dist-test-slave-zx46 krb5kdc[4860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:34.497948 4873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:20:34.498796 4873 webserver.cc:460] Webserver started at http://127.0.0.1:37159/ using document root <none> and password file <none>
I20260924 07:20:34.499343 4873 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:34.499444 4873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:34.499691 4873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:34.501765 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "2f0e3cb2b0594640ae64302939f763f6"
format_stamp: "Formatted at 2026-09-24 07:20:34 on dist-test-slave-zx46"
server_key: "20cf746d20d1d12ebe00f2a0e8a33290"
server_key_iv: "344eeb0201ed435336401226078ad0d1"
server_key_version: "encryptionkey@0"
I20260924 07:20:34.502328 4873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "2f0e3cb2b0594640ae64302939f763f6"
format_stamp: "Formatted at 2026-09-24 07:20:34 on dist-test-slave-zx46"
server_key: "20cf746d20d1d12ebe00f2a0e8a33290"
server_key_iv: "344eeb0201ed435336401226078ad0d1"
server_key_version: "encryptionkey@0"
I20260924 07:20:34.505916 4873 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:34.508833 4888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:34.510421 4873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:20:34.510540 4873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "2f0e3cb2b0594640ae64302939f763f6"
format_stamp: "Formatted at 2026-09-24 07:20:34 on dist-test-slave-zx46"
server_key: "20cf746d20d1d12ebe00f2a0e8a33290"
server_key_iv: "344eeb0201ed435336401226078ad0d1"
server_key_version: "encryptionkey@0"
I20260924 07:20:34.510659 4873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:34.515038 4873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:34.517710 4873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:34.517848 4873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:34.525053 4873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40247
I20260924 07:20:34.525065 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40247 every 8 connection(s)
I20260924 07:20:34.525985 4873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:34.526261 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4873
I20260924 07:20:34.526417 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:34.526717 15870 external_mini_cluster.cc:1428] Setting key 0ae55e470afbfb04942ad88ac28918ba
I20260924 07:20:34.529126 4941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:34.532083 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:34.528100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:56831 (local address 127.0.0.1:40247)
0924 07:20:34.528552 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:34.528567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:34.528601 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:34.529403 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:34.529423 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:34.529699 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:34.529978 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:34.530844 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:34.530848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:34.530861 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:34.530872 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:34.530928 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:34.530947 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:34.531070 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:34.531188 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 07:20:34.531381 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":120,"threads_started":1}
I20260924 07:20:34.535008 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:34.535616 4941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6: Bootstrap starting.
I20260924 07:20:34.538414 4941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:34.538867 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:34.537243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.143:52595 (local address 127.0.0.1:40247)
0924 07:20:34.537372 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:34.537377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:34.537568 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:20:34.537673 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:34.537675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:34.537704 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:34.537766 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:34.538435 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:34.538437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:34.538439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:34.538449 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:34.538497 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:34.538514 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:34.538624 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:34.538703 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:20:34.538745 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:20:34.539288 4941 log.cc:824] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:34.541226 4941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6: No bootstrap required, opened a new log
I20260924 07:20:34.544855 4941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0e3cb2b0594640ae64302939f763f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260924 07:20:34.545151 4941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:34.545233 4941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0e3cb2b0594640ae64302939f763f6, State: Initialized, Role: FOLLOWER
I20260924 07:20:34.545625 4941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0e3cb2b0594640ae64302939f763f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260924 07:20:34.545759 4941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:34.545821 4941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:34.545913 4941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:34.546777 4941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0e3cb2b0594640ae64302939f763f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260924 07:20:34.547148 4941 leader_election.cc:302] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f0e3cb2b0594640ae64302939f763f6; no voters:
I20260924 07:20:34.547518 4941 leader_election.cc:288] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:34.547772 4951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:34.547997 4951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2f0e3cb2b0594640ae64302939f763f6, State: Running, Role: LEADER
I20260924 07:20:34.548317 4951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0e3cb2b0594640ae64302939f763f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260924 07:20:34.548761 4941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:34.550022 4953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f0e3cb2b0594640ae64302939f763f6. Latest consensus state: current_term: 1 leader_uuid: "2f0e3cb2b0594640ae64302939f763f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0e3cb2b0594640ae64302939f763f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } } }
I20260924 07:20:34.550143 4953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:34.550518 4960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:34.550472 4952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f0e3cb2b0594640ae64302939f763f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0e3cb2b0594640ae64302939f763f6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } } }
I20260924 07:20:34.551437 4952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:34.553364 4960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:34.558861 4960 catalog_manager.cc:1347] Generated new cluster ID: cdc9ba2d46b640e5ac09d4d2828f9b65
I20260924 07:20:34.558969 4960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:34.595505 4960 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:34.595830 4960 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:34.608281 4960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f0e3cb2b0594640ae64302939f763f6: Generated new TSK 0
I20260924 07:20:34.608989 4960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:34.615887 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4873
2026-09-24T07:20:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:34 dist-test-slave-zx46 krb5kdc[4972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:34 dist-test-slave-zx46 krb5kdc[4972](info): set up 2 sockets
Sep 24 07:20:34 dist-test-slave-zx46 krb5kdc[4972](info): commencing operation
krb5kdc: starting...
W20260924 07:20:35.637347 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:35 dist-test-slave-zx46 krb5kdc[4972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:35.760718 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44943
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:35219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:44943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:35.868462 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:35.868763 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:35.868837 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:35.872752 4985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:35.872848 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:35.872881 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:35.872927 4985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:35.872956 4985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:35.878018 4985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:35219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:44943
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:35.879165 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:35.880263 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:35.888844 4990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:35.889088 4985 server_base.cc:1034] running on GCE node
W20260924 07:20:35.889029 4991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:35.888842 4993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:35.889822 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:35.890872 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:35.892036 4985 hybrid_clock.cc:648] HybridClock initialized: now 1790234435892021 us; error 32 us; skew 500 ppm
Sep 24 07:20:35 dist-test-slave-zx46 krb5kdc[4972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234435, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:35.894748 4985 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:35.895560 4985 webserver.cc:460] Webserver started at http://127.15.127.190:44555/ using document root <none> and password file <none>
I20260924 07:20:35.896062 4985 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:35.896134 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:35.896354 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:35.898487 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "b9ae94d9d6e6473da537aca278bc56a0"
format_stamp: "Formatted at 2026-09-24 07:20:35 on dist-test-slave-zx46"
server_key: "657ea14dd95ce064094bfa9c9f3feca6"
server_key_iv: "58d69b84169be16f507f0b3d6f550171"
server_key_version: "encryptionkey@0"
I20260924 07:20:35.899078 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "b9ae94d9d6e6473da537aca278bc56a0"
format_stamp: "Formatted at 2026-09-24 07:20:35 on dist-test-slave-zx46"
server_key: "657ea14dd95ce064094bfa9c9f3feca6"
server_key_iv: "58d69b84169be16f507f0b3d6f550171"
server_key_version: "encryptionkey@0"
I20260924 07:20:35.902784 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:35.905153 5000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:35.906313 4985 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:35.906438 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "b9ae94d9d6e6473da537aca278bc56a0"
format_stamp: "Formatted at 2026-09-24 07:20:35 on dist-test-slave-zx46"
server_key: "657ea14dd95ce064094bfa9c9f3feca6"
server_key_iv: "58d69b84169be16f507f0b3d6f550171"
server_key_version: "encryptionkey@0"
I20260924 07:20:35.906535 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:35.933826 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:35.937250 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:35.937461 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:35.944478 4985 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:44943
I20260924 07:20:35.944502 5052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:44943 every 8 connection(s)
I20260924 07:20:35.945472 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:35.946970 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 4985
I20260924 07:20:35.947124 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:35.947438 15870 external_mini_cluster.cc:1428] Setting key 4f548b67f376ca4e2361d0b6b515c68c
I20260924 07:20:35.948520 5053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:35.955282 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0: Bootstrap starting.
Sep 24 07:20:35 dist-test-slave-zx46 krb5kdc[4972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:35.957896 5053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:35.958837 5053 log.cc:824] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:35.960713 5053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0: No bootstrap required, opened a new log
I20260924 07:20:35.962507 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:35.948807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54439 (local address 127.15.127.190:44943)
0924 07:20:35.949262 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:35.949281 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:35.949322 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:20:35.950152 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:35.950175 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:35.950580 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:35.950958 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:35.951608 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:35.953143 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:35.953993 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:35.954694 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:35.954855 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:35.957373 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:35.957379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:35.957394 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:35.957424 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:35.959400 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:35.959931 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:35.959941 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:35.959945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:35.960061 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:35.960407 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:35.960410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:35.960412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:35.960793 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:35.961062 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:35.961383 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 07:20:35.961603 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":176,"threads_started":1}
I20260924 07:20:35.964087 5053 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ae94d9d6e6473da537aca278bc56a0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44943 } }
I20260924 07:20:35.964373 5053 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:35.964427 5053 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ae94d9d6e6473da537aca278bc56a0, State: Initialized, Role: FOLLOWER
I20260924 07:20:35.964895 5053 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ae94d9d6e6473da537aca278bc56a0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44943 } }
I20260924 07:20:35.965097 5053 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:35.965164 5053 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:35.965262 5053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:35.965545 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:35.966275 5053 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ae94d9d6e6473da537aca278bc56a0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44943 } }
I20260924 07:20:35.966636 5053 leader_election.cc:302] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ae94d9d6e6473da537aca278bc56a0; no voters:
I20260924 07:20:35.966950 5053 leader_election.cc:288] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:35.967051 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:35.967242 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [term 1 LEADER]: Becoming Leader. State: Replica: b9ae94d9d6e6473da537aca278bc56a0, State: Running, Role: LEADER
I20260924 07:20:35.967620 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ae94d9d6e6473da537aca278bc56a0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44943 } }
I20260924 07:20:35.968320 5053 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:35.969496 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ae94d9d6e6473da537aca278bc56a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ae94d9d6e6473da537aca278bc56a0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44943 } } }
I20260924 07:20:35.969678 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:35.969934 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ae94d9d6e6473da537aca278bc56a0. Latest consensus state: current_term: 1 leader_uuid: "b9ae94d9d6e6473da537aca278bc56a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ae94d9d6e6473da537aca278bc56a0" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44943 } } }
I20260924 07:20:35.970079 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:35.970361 5072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:35.974018 5072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:35.977607 5056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:35.967722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35419 (local address 127.15.127.190:44943)
0924 07:20:35.967839 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:35.967843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:35.968215 (+ 372us) server_negotiation.cc:407] Connection header received
0924 07:20:35.968324 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:35.968328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:35.968407 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:35.968486 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:35.969502 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:35.970357 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:35.971568 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:35.971893 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:35.971991 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:35.973800 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:35.973804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:35.973807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:35.973832 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:35.975607 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:35.976188 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:35.976194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:35.976197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:35.976282 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:35.976647 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:35.976651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:35.976652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:35.976869 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:35.977041 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:35.977352 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 07:20:35.977454 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:20:35.979826 5072 catalog_manager.cc:1347] Generated new cluster ID: 3ee12edb2e3f4a6a94e7c83b655ff548
I20260924 07:20:35.979882 5072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:35.987921 5072 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:35.988747 5072 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:35.999017 5072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ae94d9d6e6473da537aca278bc56a0: Generated new TSK 0
I20260924 07:20:35.999738 5072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:36.004240 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 4985
W20260924 07:20:36.009640 4970 connection.cc:535] client connection to 127.15.127.190:44943 recv error: Network error: failed to read from TLS socket (remote: 127.15.127.190:44943): Connection reset by peer (error 104)
2026-09-24T07:20:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:36 dist-test-slave-zx46 krb5kdc[5084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:36 dist-test-slave-zx46 krb5kdc[5084](info): set up 2 sockets
Sep 24 07:20:36 dist-test-slave-zx46 krb5kdc[5084](info): commencing operation
krb5kdc: starting...
W20260924 07:20:37.033375 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:37 dist-test-slave-zx46 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:37.150332 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:42987
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:42987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:37.255685 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:37.255946 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:37.256004 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:37.259864 5097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:37.259912 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:37.259930 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:37.259945 5097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:37.259960 5097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:37.265172 5097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:42987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:37.266093 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:37.266996 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:37.275095 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:37.275177 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:37.275122 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:37.275365 5097 server_base.cc:1034] running on GCE node
I20260924 07:20:37.275884 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:37.276921 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:37.278088 5097 hybrid_clock.cc:648] HybridClock initialized: now 1790234437278060 us; error 41 us; skew 500 ppm
Sep 24 07:20:37 dist-test-slave-zx46 krb5kdc[5084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234437, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:37.280825 5097 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:37.281735 5097 webserver.cc:460] Webserver started at http://127.15.127.190:46883/ using document root <none> and password file <none>
I20260924 07:20:37.282238 5097 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:37.282295 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:37.282452 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:37.284410 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "c953b3e15e2940dc965ad9b28763f26c"
format_stamp: "Formatted at 2026-09-24 07:20:37 on dist-test-slave-zx46"
server_key: "cb32f6dbacab66eb5c54142bf6d6f04e"
server_key_iv: "d683f0b90d3ba13386bfbfcfd018357d"
server_key_version: "encryptionkey@0"
I20260924 07:20:37.284930 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "c953b3e15e2940dc965ad9b28763f26c"
format_stamp: "Formatted at 2026-09-24 07:20:37 on dist-test-slave-zx46"
server_key: "cb32f6dbacab66eb5c54142bf6d6f04e"
server_key_iv: "d683f0b90d3ba13386bfbfcfd018357d"
server_key_version: "encryptionkey@0"
I20260924 07:20:37.288445 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:20:37.290820 5112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:37.291971 5097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260924 07:20:37.292066 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "c953b3e15e2940dc965ad9b28763f26c"
format_stamp: "Formatted at 2026-09-24 07:20:37 on dist-test-slave-zx46"
server_key: "cb32f6dbacab66eb5c54142bf6d6f04e"
server_key_iv: "d683f0b90d3ba13386bfbfcfd018357d"
server_key_version: "encryptionkey@0"
I20260924 07:20:37.292145 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:37.304612 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:37.307756 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:37.307899 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:37.314599 5097 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:42987
I20260924 07:20:37.314675 5164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:42987 every 8 connection(s)
I20260924 07:20:37.315488 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:37.315686 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5097
I20260924 07:20:37.315847 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:37.316185 15870 external_mini_cluster.cc:1428] Setting key e118dcf186814cc1767e3e01dcfcda64
I20260924 07:20:37.318691 5165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:37.324898 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c: Bootstrap starting.
I20260924 07:20:37.325273 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:37.317675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48937 (local address 127.15.127.190:42987)
0924 07:20:37.318117 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:37.318132 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:37.318166 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:37.318854 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:37.318880 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:37.319221 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:37.319493 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:37.319874 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:37.320854 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:37.322099 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:37.322557 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:37.322678 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:37.323605 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:37.323611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:37.323622 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:37.323981 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:37.324046 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:37.324067 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:37.324164 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:37.324396 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 07:20:37.324526 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":300,"thread_start_us":139,"threads_started":1}
I20260924 07:20:37.327133 5165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:37.327842 5165 log.cc:824] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:37.328081 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:37.330034 5165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c: No bootstrap required, opened a new log
I20260924 07:20:37.334221 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:37.330193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37909 (local address 127.15.127.190:42987)
0924 07:20:37.330308 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:37.330312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:37.330525 (+ 213us) server_negotiation.cc:407] Connection header received
0924 07:20:37.330640 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:37.330642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:37.330674 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:37.330734 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:37.331017 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:37.331475 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:37.332705 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:37.332901 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:37.332973 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:37.333777 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:37.333780 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:37.333782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:37.333792 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:37.333814 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:37.333823 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:37.333891 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:37.334034 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:20:37.334062 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:20:37.334622 5165 raft_consensus.cc:348] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c953b3e15e2940dc965ad9b28763f26c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42987 } }
I20260924 07:20:37.334859 5165 raft_consensus.cc:374] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:37.334928 5165 raft_consensus.cc:729] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c953b3e15e2940dc965ad9b28763f26c, State: Initialized, Role: FOLLOWER
I20260924 07:20:37.335369 5165 consensus_queue.cc:260] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c953b3e15e2940dc965ad9b28763f26c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42987 } }
I20260924 07:20:37.335518 5165 raft_consensus.cc:388] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:37.335584 5165 raft_consensus.cc:482] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:37.335675 5165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:37.336508 5165 raft_consensus.cc:504] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c953b3e15e2940dc965ad9b28763f26c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42987 } }
I20260924 07:20:37.336835 5165 leader_election.cc:302] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c953b3e15e2940dc965ad9b28763f26c; no voters:
I20260924 07:20:37.337157 5165 leader_election.cc:288] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:37.337296 5175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:37.337538 5175 raft_consensus.cc:686] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [term 1 LEADER]: Becoming Leader. State: Replica: c953b3e15e2940dc965ad9b28763f26c, State: Running, Role: LEADER
I20260924 07:20:37.337872 5175 consensus_queue.cc:237] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c953b3e15e2940dc965ad9b28763f26c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42987 } }
I20260924 07:20:37.338318 5165 sys_catalog.cc:564] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:37.339449 5176 sys_catalog.cc:455] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c953b3e15e2940dc965ad9b28763f26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c953b3e15e2940dc965ad9b28763f26c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42987 } } }
I20260924 07:20:37.339576 5176 sys_catalog.cc:458] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:37.339751 5177 sys_catalog.cc:455] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c953b3e15e2940dc965ad9b28763f26c. Latest consensus state: current_term: 1 leader_uuid: "c953b3e15e2940dc965ad9b28763f26c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c953b3e15e2940dc965ad9b28763f26c" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 42987 } } }
I20260924 07:20:37.339917 5177 sys_catalog.cc:458] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:37.340166 5186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:37.342798 5186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:37.347872 5186 catalog_manager.cc:1347] Generated new cluster ID: bec47b49459947beadb4f8472fbf407a
I20260924 07:20:37.347931 5186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:37.371733 5186 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:37.372849 5186 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:37.380034 5186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c953b3e15e2940dc965ad9b28763f26c: Generated new TSK 0
I20260924 07:20:37.380764 5186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:37.392831 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5097
W20260924 07:20:37.398234 5082 connection.cc:535] client connection to 127.15.127.190:42987 recv error: Network error: failed to read from TLS socket (remote: 127.15.127.190:42987): Connection reset by peer (error 104)
2026-09-24T07:20:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:37 dist-test-slave-zx46 krb5kdc[5196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:37 dist-test-slave-zx46 krb5kdc[5196](info): set up 2 sockets
Sep 24 07:20:37 dist-test-slave-zx46 krb5kdc[5196](info): commencing operation
krb5kdc: starting...
W20260924 07:20:38.425513 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:38 dist-test-slave-zx46 krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:38.550022 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:39731
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:35995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:38.661402 5209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:38.661653 5209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:38.661732 5209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:38.665874 5209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:38.665936 5209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:38.665956 5209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:38.666009 5209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:38.666034 5209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:38.671150 5209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:35995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:39731
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:38.672188 5209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:38.673305 5209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:38.681556 5217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:38.681566 5215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:38.681586 5214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:38.681886 5209 server_base.cc:1034] running on GCE node
I20260924 07:20:38.682442 5209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:38.683471 5209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:38.684628 5209 hybrid_clock.cc:648] HybridClock initialized: now 1790234438684599 us; error 41 us; skew 500 ppm
Sep 24 07:20:38 dist-test-slave-zx46 krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234438, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:38.687309 5209 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:38.688200 5209 webserver.cc:460] Webserver started at http://127.15.127.190:38779/ using document root <none> and password file <none>
I20260924 07:20:38.688752 5209 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:38.688812 5209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:38.689101 5209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:38.691083 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "a77f01a619eb4361833d80ab3fad47ca"
format_stamp: "Formatted at 2026-09-24 07:20:38 on dist-test-slave-zx46"
server_key: "b08680fc7784e3807088f3ae4b8b1bec"
server_key_iv: "7524b215f513e716938113b7aa04e19d"
server_key_version: "encryptionkey@0"
I20260924 07:20:38.691645 5209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "a77f01a619eb4361833d80ab3fad47ca"
format_stamp: "Formatted at 2026-09-24 07:20:38 on dist-test-slave-zx46"
server_key: "b08680fc7784e3807088f3ae4b8b1bec"
server_key_iv: "7524b215f513e716938113b7aa04e19d"
server_key_version: "encryptionkey@0"
I20260924 07:20:38.695235 5209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:38.697590 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:38.698769 5209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:38.698932 5209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "a77f01a619eb4361833d80ab3fad47ca"
format_stamp: "Formatted at 2026-09-24 07:20:38 on dist-test-slave-zx46"
server_key: "b08680fc7784e3807088f3ae4b8b1bec"
server_key_iv: "7524b215f513e716938113b7aa04e19d"
server_key_version: "encryptionkey@0"
I20260924 07:20:38.699048 5209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:38.703519 5209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:38.706314 5209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:38.706511 5209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:38.713410 5209 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:39731
I20260924 07:20:38.713428 5276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:39731 every 8 connection(s)
I20260924 07:20:38.714288 5209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:38.716150 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5209
I20260924 07:20:38.716301 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:38.716612 15870 external_mini_cluster.cc:1428] Setting key 9aacaad65daec9aa5aa2d98461a131c6
I20260924 07:20:38.717216 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:38.721601 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:38.717948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60487 (local address 127.15.127.190:39731)
0924 07:20:38.718391 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:38.718405 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:38.718439 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:20:38.719071 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:38.719087 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:38.719341 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:38.719686 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:38.720431 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:38.720436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:38.720447 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:38.720457 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:38.720507 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:38.720525 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:38.720672 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:38.720741 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:20:38.720882 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":140,"threads_started":1}
I20260924 07:20:38.723464 5277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca: Bootstrap starting.
I20260924 07:20:38.724371 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:38.725909 5277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:38.726727 5277 log.cc:824] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:38.728093 5280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:38.726409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54137 (local address 127.15.127.190:39731)
0924 07:20:38.726539 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:38.726544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:38.726739 (+ 195us) server_negotiation.cc:407] Connection header received
0924 07:20:38.726870 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:38.726875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:38.726909 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:38.726984 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:38.727730 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:38.727733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:38.727736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:38.727747 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:38.727772 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:38.727784 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:38.727870 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:38.727929 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:20:38.727962 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:20:38.728796 5277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca: No bootstrap required, opened a new log
I20260924 07:20:38.732180 5277 raft_consensus.cc:348] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f01a619eb4361833d80ab3fad47ca" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39731 } }
I20260924 07:20:38.732389 5277 raft_consensus.cc:374] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:38.732484 5277 raft_consensus.cc:729] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a77f01a619eb4361833d80ab3fad47ca, State: Initialized, Role: FOLLOWER
I20260924 07:20:38.732964 5277 consensus_queue.cc:260] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f01a619eb4361833d80ab3fad47ca" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39731 } }
I20260924 07:20:38.733167 5277 raft_consensus.cc:388] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:38.733237 5277 raft_consensus.cc:482] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:38.733333 5277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:38.734330 5277 raft_consensus.cc:504] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f01a619eb4361833d80ab3fad47ca" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39731 } }
I20260924 07:20:38.734673 5277 leader_election.cc:302] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a77f01a619eb4361833d80ab3fad47ca; no voters:
I20260924 07:20:38.734977 5277 leader_election.cc:288] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:38.735101 5287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:38.735328 5287 raft_consensus.cc:686] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [term 1 LEADER]: Becoming Leader. State: Replica: a77f01a619eb4361833d80ab3fad47ca, State: Running, Role: LEADER
I20260924 07:20:38.735658 5287 consensus_queue.cc:237] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f01a619eb4361833d80ab3fad47ca" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39731 } }
I20260924 07:20:38.736208 5277 sys_catalog.cc:564] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:38.737509 5288 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a77f01a619eb4361833d80ab3fad47ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f01a619eb4361833d80ab3fad47ca" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39731 } } }
I20260924 07:20:38.737524 5289 sys_catalog.cc:455] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a77f01a619eb4361833d80ab3fad47ca. Latest consensus state: current_term: 1 leader_uuid: "a77f01a619eb4361833d80ab3fad47ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a77f01a619eb4361833d80ab3fad47ca" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 39731 } } }
I20260924 07:20:38.737636 5289 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:38.737635 5288 sys_catalog.cc:458] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:38.738821 5297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:38.741506 5297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:38.747026 5297 catalog_manager.cc:1347] Generated new cluster ID: 6f103146fc4142a29d4940e56bb0ba0b
I20260924 07:20:38.747275 5297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:38.758889 5297 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:38.759320 5297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:38.766498 5297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a77f01a619eb4361833d80ab3fad47ca: Generated new TSK 0
I20260924 07:20:38.767253 5297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:38.772339 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5209
2026-09-24T07:20:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1378 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:38 dist-test-slave-zx46 krb5kdc[5308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:38 dist-test-slave-zx46 krb5kdc[5308](info): set up 2 sockets
Sep 24 07:20:38 dist-test-slave-zx46 krb5kdc[5308](info): commencing operation
krb5kdc: starting...
W20260924 07:20:39.805323 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:39 dist-test-slave-zx46 krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:39.924032 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:32841
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:38395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:32841
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:40.032895 5321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:40.033245 5321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:40.033324 5321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:40.037278 5321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:40.037355 5321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:40.037390 5321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:40.037421 5321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:40.037451 5321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:40.042590 5321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:38395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:32841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:32841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:40.043715 5321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:40.044759 5321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:40.053107 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:40.053208 5326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:40.053148 5329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:40.053318 5321 server_base.cc:1034] running on GCE node
I20260924 07:20:40.053834 5321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:40.054867 5321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:40.056035 5321 hybrid_clock.cc:648] HybridClock initialized: now 1790234440056017 us; error 30 us; skew 500 ppm
Sep 24 07:20:40 dist-test-slave-zx46 krb5kdc[5308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234440, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:40.065858 5321 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:40.066736 5321 webserver.cc:460] Webserver started at http://127.15.127.190:43923/ using document root <none> and password file <none>
I20260924 07:20:40.067221 5321 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:40.067273 5321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:40.067421 5321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:40.069502 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "cfd0c607506c4f9186b5818f7e01234d"
format_stamp: "Formatted at 2026-09-24 07:20:40 on dist-test-slave-zx46"
server_key: "5a586d040ba2f6debd547ad23eb03245"
server_key_iv: "ce5fe47c701dc695279cc292a2ff013b"
server_key_version: "encryptionkey@0"
I20260924 07:20:40.069983 5321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "cfd0c607506c4f9186b5818f7e01234d"
format_stamp: "Formatted at 2026-09-24 07:20:40 on dist-test-slave-zx46"
server_key: "5a586d040ba2f6debd547ad23eb03245"
server_key_iv: "ce5fe47c701dc695279cc292a2ff013b"
server_key_version: "encryptionkey@0"
I20260924 07:20:40.073479 5321 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 07:20:40.075840 5336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:40.077097 5321 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:40.077221 5321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "cfd0c607506c4f9186b5818f7e01234d"
format_stamp: "Formatted at 2026-09-24 07:20:40 on dist-test-slave-zx46"
server_key: "5a586d040ba2f6debd547ad23eb03245"
server_key_iv: "ce5fe47c701dc695279cc292a2ff013b"
server_key_version: "encryptionkey@0"
I20260924 07:20:40.077324 5321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:40.091967 5321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:40.095180 5321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:40.095376 5321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:40.101956 5321 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:32841
I20260924 07:20:40.101982 5388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:32841 every 8 connection(s)
I20260924 07:20:40.102862 5321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:40.105619 5389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:40.109920 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5321
I20260924 07:20:40.110091 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:40.110357 15870 external_mini_cluster.cc:1428] Setting key 7072472e2188dcf4977e50f8149a186f
I20260924 07:20:40.111450 5389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d: Bootstrap starting.
I20260924 07:20:40.114213 5389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:40.115082 5389 log.cc:824] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:40.117050 5389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d: No bootstrap required, opened a new log
Sep 24 07:20:40 dist-test-slave-zx46 krb5kdc[5308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:40.121294 5389 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd0c607506c4f9186b5818f7e01234d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 32841 } }
I20260924 07:20:40.121585 5389 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:40.121666 5389 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd0c607506c4f9186b5818f7e01234d, State: Initialized, Role: FOLLOWER
I20260924 07:20:40.122166 5389 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd0c607506c4f9186b5818f7e01234d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 32841 } }
I20260924 07:20:40.122327 5389 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:40.122398 5389 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:40.122500 5389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:40.123672 5389 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd0c607506c4f9186b5818f7e01234d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 32841 } }
I20260924 07:20:40.124070 5389 leader_election.cc:302] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfd0c607506c4f9186b5818f7e01234d; no voters:
I20260924 07:20:40.124406 5389 leader_election.cc:288] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:40.124533 5394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:40.124783 5394 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [term 1 LEADER]: Becoming Leader. State: Replica: cfd0c607506c4f9186b5818f7e01234d, State: Running, Role: LEADER
I20260924 07:20:40.125169 5394 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd0c607506c4f9186b5818f7e01234d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 32841 } }
I20260924 07:20:40.125464 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:40.111781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55769 (local address 127.15.127.190:32841)
0924 07:20:40.112223 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:40.112241 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:40.112284 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:20:40.113188 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:40.113212 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:40.113639 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:40.113992 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:40.114763 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:40.116101 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:40.116953 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:40.117693 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:40.117854 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:40.120217 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:40.120224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:40.120240 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:40.120271 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:40.122240 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:40.122748 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:40.122755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:40.122760 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:40.122890 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:40.123166 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:40.123169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:40.123171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:40.123718 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:40.123897 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:40.124177 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 07:20:40.124499 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":126,"threads_started":1}
I20260924 07:20:40.125748 5389 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:40.126870 5396 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfd0c607506c4f9186b5818f7e01234d. Latest consensus state: current_term: 1 leader_uuid: "cfd0c607506c4f9186b5818f7e01234d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd0c607506c4f9186b5818f7e01234d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 32841 } } }
I20260924 07:20:40.126940 5395 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfd0c607506c4f9186b5818f7e01234d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfd0c607506c4f9186b5818f7e01234d" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 32841 } } }
I20260924 07:20:40.126996 5396 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:40.127007 5395 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:40.128652 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 07:20:40.130589 5409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:40.130662 5409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:40.130771 5403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:40.131735 5403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:40.138079 5403 catalog_manager.cc:1347] Generated new cluster ID: a2b5862fce8c4758ade18ff18c7aae3f
I20260924 07:20:40.138139 5403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:40.139503 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:40.131122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34413 (local address 127.15.127.190:32841)
0924 07:20:40.131226 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:40.131230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:40.131473 (+ 243us) server_negotiation.cc:407] Connection header received
0924 07:20:40.131577 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:40.131580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:40.131625 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:40.131708 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:40.132506 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:40.133386 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:40.134238 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:40.134481 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:40.134607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:40.136019 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:40.136022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:40.136026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:40.136050 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:40.137707 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:40.138231 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:40.138234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:40.138236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:40.138318 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:40.138631 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:40.138634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:40.138636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:40.138843 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:40.138973 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:40.139233 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 07:20:40.139351 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:20:40.159675 5403 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:40.160669 5403 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:40.165629 5403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfd0c607506c4f9186b5818f7e01234d: Generated new TSK 0
I20260924 07:20:40.166301 5403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:40.172405 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5321
2026-09-24T07:20:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:40 dist-test-slave-zx46 krb5kdc[5420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:40 dist-test-slave-zx46 krb5kdc[5420](info): set up 2 sockets
Sep 24 07:20:40 dist-test-slave-zx46 krb5kdc[5420](info): commencing operation
krb5kdc: starting...
W20260924 07:20:41.201323 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:41 dist-test-slave-zx46 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:41.319681 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44463
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:37803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:44463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:41.427084 5433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:41.427348 5433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:41.427389 5433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:41.431345 5433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:41.431396 5433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:41.431416 5433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:41.431433 5433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:41.431493 5433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:41.436579 5433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:37803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:44463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:44463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:41.437633 5433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:41.438658 5433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:41.446867 5439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:41.446906 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:41.446906 5438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:41.447108 5433 server_base.cc:1034] running on GCE node
I20260924 07:20:41.447693 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:41.448732 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:41.449898 5433 hybrid_clock.cc:648] HybridClock initialized: now 1790234441449862 us; error 46 us; skew 500 ppm
Sep 24 07:20:41 dist-test-slave-zx46 krb5kdc[5420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234441, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:41.452651 5433 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:41.453554 5433 webserver.cc:460] Webserver started at http://127.15.127.190:36683/ using document root <none> and password file <none>
I20260924 07:20:41.454051 5433 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:41.454140 5433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:41.454341 5433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:41.456295 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "02aad9232c5e42289866a1c8ff274edc"
format_stamp: "Formatted at 2026-09-24 07:20:41 on dist-test-slave-zx46"
server_key: "dcb6750591e35c5a950e232ab2f78b1d"
server_key_iv: "1aa6012f9b3176339c3e69658b81b075"
server_key_version: "encryptionkey@0"
I20260924 07:20:41.456802 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "02aad9232c5e42289866a1c8ff274edc"
format_stamp: "Formatted at 2026-09-24 07:20:41 on dist-test-slave-zx46"
server_key: "dcb6750591e35c5a950e232ab2f78b1d"
server_key_iv: "1aa6012f9b3176339c3e69658b81b075"
server_key_version: "encryptionkey@0"
I20260924 07:20:41.460367 5433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 07:20:41.462670 5448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:41.463860 5433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:20:41.464008 5433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "02aad9232c5e42289866a1c8ff274edc"
format_stamp: "Formatted at 2026-09-24 07:20:41 on dist-test-slave-zx46"
server_key: "dcb6750591e35c5a950e232ab2f78b1d"
server_key_iv: "1aa6012f9b3176339c3e69658b81b075"
server_key_version: "encryptionkey@0"
I20260924 07:20:41.464118 5433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:41.477687 5433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:41.480600 5433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:41.480738 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:41.487365 5433 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:44463
I20260924 07:20:41.487401 5500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:44463 every 8 connection(s)
I20260924 07:20:41.488195 5433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:41.490809 5501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:41.494884 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5433
I20260924 07:20:41.495023 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:41.495316 15870 external_mini_cluster.cc:1428] Setting key f69c5f2fbbc97670bf24090098dda137
I20260924 07:20:41.496353 5501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc: Bootstrap starting.
I20260924 07:20:41.498659 5501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:41.499373 5501 log.cc:824] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:41.501194 5501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc: No bootstrap required, opened a new log
I20260924 07:20:41.503892 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:41.496685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37995 (local address 127.15.127.190:44463)
0924 07:20:41.497151 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:41.497166 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:41.497199 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:20:41.497842 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:41.497857 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:41.498185 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:41.498532 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:41.498914 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:41.499905 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:41.500768 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:41.501326 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:41.501513 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:41.502328 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:41.502334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:41.502354 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:41.502713 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:41.502764 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:41.502785 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:41.502889 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:41.503019 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 07:20:41.503235 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":132,"threads_started":1}
I20260924 07:20:41.505098 5501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aad9232c5e42289866a1c8ff274edc" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44463 } }
I20260924 07:20:41.505354 5501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:41.505429 5501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02aad9232c5e42289866a1c8ff274edc, State: Initialized, Role: FOLLOWER
I20260924 07:20:41.505899 5501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aad9232c5e42289866a1c8ff274edc" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44463 } }
I20260924 07:20:41.506086 5501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:41.506166 5501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:41.506263 5501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:41.506733 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:41.507354 5501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aad9232c5e42289866a1c8ff274edc" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44463 } }
I20260924 07:20:41.507714 5501 leader_election.cc:302] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02aad9232c5e42289866a1c8ff274edc; no voters:
I20260924 07:20:41.508045 5501 leader_election.cc:288] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:41.508157 5507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:41.508428 5507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [term 1 LEADER]: Becoming Leader. State: Replica: 02aad9232c5e42289866a1c8ff274edc, State: Running, Role: LEADER
I20260924 07:20:41.508858 5507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aad9232c5e42289866a1c8ff274edc" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44463 } }
I20260924 07:20:41.509425 5501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:41.511152 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02aad9232c5e42289866a1c8ff274edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aad9232c5e42289866a1c8ff274edc" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44463 } } }
I20260924 07:20:41.511318 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:41.512061 5512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02aad9232c5e42289866a1c8ff274edc. Latest consensus state: current_term: 1 leader_uuid: "02aad9232c5e42289866a1c8ff274edc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02aad9232c5e42289866a1c8ff274edc" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 44463 } } }
I20260924 07:20:41.512247 5512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:41.513664 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:41.508907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52413 (local address 127.15.127.190:44463)
0924 07:20:41.509027 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:41.509031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:41.509195 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:20:41.509346 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:41.509349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:41.509389 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:41.509470 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:41.509798 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:41.510567 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:41.511739 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:41.512032 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:41.512145 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:41.513029 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:41.513033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:41.513036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:41.513049 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:41.513076 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:41.513090 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:41.513163 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:41.513488 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 07:20:41.513547 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260924 07:20:41.514874 5525 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:41.514935 5525 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:41.515012 5526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:41.515687 5526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:41.520498 5526 catalog_manager.cc:1347] Generated new cluster ID: 80f2875a96d74c48947d75ba617fe731
I20260924 07:20:41.520558 5526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:41.527029 5526 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:41.527896 5526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:41.537248 5526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02aad9232c5e42289866a1c8ff274edc: Generated new TSK 0
I20260924 07:20:41.537993 5526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:41.547508 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5433
2026-09-24T07:20:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:41 dist-test-slave-zx46 krb5kdc[5532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:41 dist-test-slave-zx46 krb5kdc[5532](info): set up 2 sockets
Sep 24 07:20:41 dist-test-slave-zx46 krb5kdc[5532](info): commencing operation
krb5kdc: starting...
W20260924 07:20:42.565331 15870 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:42 dist-test-slave-zx46 krb5kdc[5532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:42.684289 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:40399
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:46771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:40399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:42.790578 5545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:42.790876 5545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:42.790962 5545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:42.794991 5545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:42.795095 5545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:42.795135 5545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:42.795189 5545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:42.795222 5545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:42.800455 5545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:46771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:40399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:40399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:42.801591 5545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:42.802634 5545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:42.811685 5553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:42.811690 5550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:42.811690 5551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:42.811760 5545 server_base.cc:1034] running on GCE node
I20260924 07:20:42.812453 5545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:42.813649 5545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:42.814815 5545 hybrid_clock.cc:648] HybridClock initialized: now 1790234442814800 us; error 28 us; skew 500 ppm
Sep 24 07:20:42 dist-test-slave-zx46 krb5kdc[5532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234442, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:42.817577 5545 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:42.818423 5545 webserver.cc:460] Webserver started at http://127.15.127.190:42067/ using document root <none> and password file <none>
I20260924 07:20:42.818966 5545 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:42.819031 5545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:42.819259 5545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:42.821373 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "6866ed987fd94185bb6f696a478e9bf3"
format_stamp: "Formatted at 2026-09-24 07:20:42 on dist-test-slave-zx46"
server_key: "c5c9c7ee54cf81bf637960fffff7c1a3"
server_key_iv: "5ca875a7aa31b73dca81ff7304669eea"
server_key_version: "encryptionkey@0"
I20260924 07:20:42.822000 5545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "6866ed987fd94185bb6f696a478e9bf3"
format_stamp: "Formatted at 2026-09-24 07:20:42 on dist-test-slave-zx46"
server_key: "c5c9c7ee54cf81bf637960fffff7c1a3"
server_key_iv: "5ca875a7aa31b73dca81ff7304669eea"
server_key_version: "encryptionkey@0"
I20260924 07:20:42.825706 5545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 07:20:42.828068 5560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:42.829258 5545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:42.829393 5545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "6866ed987fd94185bb6f696a478e9bf3"
format_stamp: "Formatted at 2026-09-24 07:20:42 on dist-test-slave-zx46"
server_key: "c5c9c7ee54cf81bf637960fffff7c1a3"
server_key_iv: "5ca875a7aa31b73dca81ff7304669eea"
server_key_version: "encryptionkey@0"
I20260924 07:20:42.829499 5545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:42.834084 5545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:42.836850 5545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:42.837121 5545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:42.843816 5545 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:40399
I20260924 07:20:42.843884 5612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:40399 every 8 connection(s)
I20260924 07:20:42.844779 5545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:42.847558 5613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:42.850570 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5545
I20260924 07:20:42.850714 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:42.850973 15870 external_mini_cluster.cc:1428] Setting key efe3edc47ee5ab9549534ad5d5ddeb89
I20260924 07:20:42.853617 5613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3: Bootstrap starting.
I20260924 07:20:42.856209 5613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:42.856226 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:42.852294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42385 (local address 127.15.127.190:40399)
0924 07:20:42.852755 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:42.852771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:42.852807 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:20:42.853483 (+ 676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:42.853502 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:42.853838 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:42.854134 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:42.855016 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:42.855020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:42.855032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:42.855042 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:42.855095 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:42.855115 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:42.855220 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:42.855347 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:20:42.855524 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":142,"threads_started":1}
I20260924 07:20:42.857203 5613 log.cc:824] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:42.859313 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:42.859416 5613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3: No bootstrap required, opened a new log
I20260924 07:20:42.862874 5616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:42.861311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49989 (local address 127.15.127.190:40399)
0924 07:20:42.861439 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:42.861443 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:42.861571 (+ 128us) server_negotiation.cc:407] Connection header received
0924 07:20:42.861660 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:42.861663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:42.861695 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:42.861769 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:42.862466 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:42.862472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:42.862476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:20:42.862491 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:42.862526 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:20:42.862542 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:42.862646 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:42.862705 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:20:42.862750 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:20:42.863745 5613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6866ed987fd94185bb6f696a478e9bf3" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40399 } }
I20260924 07:20:42.863957 5613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:20:42.864001 5613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6866ed987fd94185bb6f696a478e9bf3, State: Initialized, Role: FOLLOWER
I20260924 07:20:42.864408 5613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6866ed987fd94185bb6f696a478e9bf3" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40399 } }
I20260924 07:20:42.864540 5613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:20:42.864588 5613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:20:42.864670 5613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:42.865588 5613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6866ed987fd94185bb6f696a478e9bf3" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40399 } }
I20260924 07:20:42.865882 5613 leader_election.cc:302] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6866ed987fd94185bb6f696a478e9bf3; no voters:
I20260924 07:20:42.866122 5613 leader_election.cc:288] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:20:42.866251 5623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:42.866477 5623 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [term 1 LEADER]: Becoming Leader. State: Replica: 6866ed987fd94185bb6f696a478e9bf3, State: Running, Role: LEADER
I20260924 07:20:42.866797 5623 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6866ed987fd94185bb6f696a478e9bf3" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40399 } }
I20260924 07:20:42.867383 5613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:42.868570 5624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6866ed987fd94185bb6f696a478e9bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6866ed987fd94185bb6f696a478e9bf3" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40399 } } }
I20260924 07:20:42.868721 5624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:42.868956 5625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6866ed987fd94185bb6f696a478e9bf3. Latest consensus state: current_term: 1 leader_uuid: "6866ed987fd94185bb6f696a478e9bf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6866ed987fd94185bb6f696a478e9bf3" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 40399 } } }
I20260924 07:20:42.869109 5632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:42.869168 5625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:42.872325 5632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:42.878079 5632 catalog_manager.cc:1347] Generated new cluster ID: bd6ce13b033f4f22ad19fb6c65e66e74
I20260924 07:20:42.878139 5632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:42.887712 5632 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:42.888044 5632 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:42.897388 5632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6866ed987fd94185bb6f696a478e9bf3: Generated new TSK 0
I20260924 07:20:42.898211 5632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:20:42.911830 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5545
2026-09-24T07:20:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1363 ms)
[----------] 18 tests from AuthTokenIssuingTest (25031 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:42 dist-test-slave-zx46 krb5kdc[5644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:42 dist-test-slave-zx46 krb5kdc[5644](info): set up 2 sockets
Sep 24 07:20:42 dist-test-slave-zx46 krb5kdc[5644](info): commencing operation
krb5kdc: starting...
W20260924 07:20:43.933513 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:44.062211 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36611
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:36611,127.15.127.189:46377,127.15.127.188:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:44.168927 5657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:20:44.169209 5657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:44.169230 5657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:44.169266 5657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:44.173071 5657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:44.173117 5657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:44.173136 5657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:44.173151 5657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:44.173166 5657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:44.178071 5657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:36611,127.15.127.189:46377,127.15.127.188:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:36611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:44.179101 5657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:44.179997 5657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:44.188117 5665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.188208 5663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.188107 5662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:44.188318 5657 server_base.cc:1034] running on GCE node
I20260924 07:20:44.188901 5657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:44.189985 5657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:44.191195 5657 hybrid_clock.cc:648] HybridClock initialized: now 1790234444191169 us; error 49 us; skew 500 ppm
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:44.193898 5657 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:44.194728 5657 webserver.cc:460] Webserver started at http://127.15.127.190:33739/ using document root <none> and password file <none>
I20260924 07:20:44.195287 5657 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:44.195423 5657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:44.195657 5657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:44.197647 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "a35eade882e64bd9a86598973d155250"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "0fa43fdc3cf0f4373a1db910babfb6b7"
server_key_iv: "1e0715c1d5d3e05ad96932c37467a052"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.198202 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "a35eade882e64bd9a86598973d155250"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "0fa43fdc3cf0f4373a1db910babfb6b7"
server_key_iv: "1e0715c1d5d3e05ad96932c37467a052"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.201786 5657 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:44.204145 5672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:44.205330 5657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 07:20:44.205472 5657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "a35eade882e64bd9a86598973d155250"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "0fa43fdc3cf0f4373a1db910babfb6b7"
server_key_iv: "1e0715c1d5d3e05ad96932c37467a052"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.205585 5657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:44.216948 5657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:44.219954 5657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:44.220185 5657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:44.226513 5657 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:36611
I20260924 07:20:44.226814 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:36611 every 8 connection(s)
I20260924 07:20:44.227524 5657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:44.228689 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5657
I20260924 07:20:44.228818 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:44.229157 15870 external_mini_cluster.cc:1428] Setting key 258e15f616dade1d1037933a90959c9d
I20260924 07:20:44.230773 5725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:44.234588 5725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.189@KRBTEST.COM" created.
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:44.249769 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.237514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33098 (local address 127.15.127.190:36611)
0924 07:20:44.237803 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.237835 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.237880 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:20:44.238358 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.238362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.238534 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.238683 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.239433 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.240695 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.242120 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.242504 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.242623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.244960 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.244969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.244989 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.245071 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.246667 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.247173 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.247179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.247180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.247289 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.247614 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.247617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.247619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.247968 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.248117 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.248485 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.248732 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260924 07:20:44.249769 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.236599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36611 (local address 127.0.0.1:33098)
0924 07:20:44.237110 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.237161 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.238090 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.238711 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.238715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.239017 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.239247 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.239258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.240824 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.240827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.241992 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.242006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.242643 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.242646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.242741 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.243339 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.243355 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.244815 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.246808 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.246815 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.246825 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.247062 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.247396 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.247399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.247401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.247515 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.248100 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.248106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.248240 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.248405 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.248780 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":117,"thread_start_us":175,"threads_started":1}
I20260924 07:20:44.252183 5725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:44.252919 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.252666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47088)
0924 07:20:44.252788 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.252860 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 07:20:44.253211 5673 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
I20260924 07:20:44.254067 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.253877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47098)
0924 07:20:44.253972 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.254017 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 07:20:44.254243 5725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:46377: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab.
Entry for principal kudu/127.15.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab.
I20260924 07:20:44.256199 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:46377
--webserver_interface=127.15.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab
--principal=kudu/127.15.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:36611,127.15.127.189:46377,127.15.127.188:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:44.303567 5725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } attempt: 1
I20260924 07:20:44.304533 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.304212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47114)
0924 07:20:44.304379 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.304442 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 07:20:44.305310 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.305126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47116)
0924 07:20:44.305219 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.305263 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 07:20:44.305465 5725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:46377: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
W20260924 07:20:44.362524 5732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:20:44.362790 5732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:44.362829 5732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:44.362865 5732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:44.366906 5732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:44.366958 5732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:44.366976 5732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:44.366998 5732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:44.367013 5732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:44.371800 5732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:36611,127.15.127.189:46377,127.15.127.188:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab
--principal=kudu/127.15.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:46377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:44.372800 5732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:44.373638 5725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } attempt: 2
I20260924 07:20:44.373872 5732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:20:44.374435 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.374158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47128)
0924 07:20:44.374303 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.374362 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:20:44.375156 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.374979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47144)
0924 07:20:44.375069 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.375109 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:20:44.375304 5725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:46377: Network error: Client connection negotiation failed: client connection to 127.15.127.189:46377: connect: Connection refused (error 111)
W20260924 07:20:44.382191 5738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.382189 5740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.382189 5737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:44.382364 5732 server_base.cc:1034] running on GCE node
I20260924 07:20:44.382987 5732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:44.384044 5732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:44.385228 5732 hybrid_clock.cc:648] HybridClock initialized: now 1790234444385204 us; error 39 us; skew 500 ppm
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:44.387775 5732 init.cc:377] Logged in from keytab as kudu/127.15.127.189@KRBTEST.COM (short username kudu)
I20260924 07:20:44.388531 5732 webserver.cc:460] Webserver started at http://127.15.127.189:46699/ using document root <none> and password file <none>
I20260924 07:20:44.389098 5732 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:44.389185 5732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:44.389392 5732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:44.391255 5732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data/instance:
uuid: "c6b23eee5a194834ae2b0724c5890b7b"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "6b3aca4886a6ce3f99fac70e905b3960"
server_key_iv: "7b55a0e855def5b88f205eeb3c5cdeee"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.391793 5732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal/instance:
uuid: "c6b23eee5a194834ae2b0724c5890b7b"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "6b3aca4886a6ce3f99fac70e905b3960"
server_key_iv: "7b55a0e855def5b88f205eeb3c5cdeee"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.395424 5732 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:44.397622 5747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:44.398749 5732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:44.398893 5732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal
uuid: "c6b23eee5a194834ae2b0724c5890b7b"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "6b3aca4886a6ce3f99fac70e905b3960"
server_key_iv: "7b55a0e855def5b88f205eeb3c5cdeee"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.399003 5732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:44.414355 5732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:44.417666 5732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:44.417869 5732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:44.424883 5732 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.189:46377
I20260924 07:20:44.425177 5799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.189:46377 every 8 connection(s)
I20260924 07:20:44.425876 5732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
I20260924 07:20:44.428572 5800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:44.432155 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5732
I20260924 07:20:44.432256 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-1/wal/instance
I20260924 07:20:44.432564 15870 external_mini_cluster.cc:1428] Setting key 4110e062ac8ce415b3d0ed24ba71134a
I20260924 07:20:44.432566 5800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.127.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.188@KRBTEST.COM" created.
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:44.448329 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.434526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.15.127.190:36611)
0924 07:20:44.434699 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.434707 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.435476 (+ 769us) server_negotiation.cc:407] Connection header received
0924 07:20:44.436475 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.436480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.436546 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.436653 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.437825 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.438700 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.439768 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.440169 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.440293 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.444334 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.444338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.444340 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.444364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.446114 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.446587 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.446593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.446598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.446712 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.447030 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.447036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.447040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.447308 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.447419 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.447928 (+ 509us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.448093 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 07:20:44.448921 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.434858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36611 (local address 127.0.0.1:33108)
0924 07:20:44.435314 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.435365 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.436194 (+ 829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.436792 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.436798 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.437230 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.437677 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.437688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.438825 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.438828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.439638 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.439645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.440374 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.440378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.440521 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.441168 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.441183 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.444146 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.446222 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.446226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.446239 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.446477 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.446803 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.446805 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.446807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.446929 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.447418 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.447423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.447664 (+ 241us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.447817 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.448070 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":135,"threads_started":1}
I20260924 07:20:44.450348 5800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:20:44.459228 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.450808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47150)
0924 07:20:44.450915 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.450929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.451002 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.451565 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.451567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.451693 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.451776 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.451781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.452699 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.452702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.453477 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.453496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.454169 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.454171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.454240 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.454719 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.454731 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.455939 (+ 1208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.457732 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.457735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.457736 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.457938 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.458272 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.458274 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.458275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.458348 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.458820 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.458822 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.458897 (+ 75us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.458980 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.459075 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:20:44.459394 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.450928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47150 (local address 127.15.127.189:46377)
0924 07:20:44.451237 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.451249 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.451265 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:20:44.451293 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.451297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.451439 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.451542 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.451875 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.452580 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.453599 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.454067 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.454154 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.456059 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.456067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.456078 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.456098 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.457632 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.458088 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.458097 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.458102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.458191 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.458426 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.458428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.458429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.458726 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.458828 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.459067 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.459178 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab.
Entry for principal kudu/127.15.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab.
I20260924 07:20:44.460796 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:34377
--webserver_interface=127.15.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab
--principal=kudu/127.15.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:41409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:36611,127.15.127.189:46377,127.15.127.188:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:44.461381 5800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:44.462065 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.461823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46940)
0924 07:20:44.461939 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.462015 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 07:20:44.462276 5750 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
I20260924 07:20:44.463023 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.462820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46956)
0924 07:20:44.462925 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.462971 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 07:20:44.463204 5800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:34377: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
I20260924 07:20:44.466459 5725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } attempt: 3
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:20:44.475580 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.466929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47166)
0924 07:20:44.467057 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.467072 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.467198 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.467434 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.467437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.467608 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.467703 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.467709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.468446 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.468450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.469602 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.469611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.470523 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.470527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.470633 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.471171 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.471182 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.472429 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.474095 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.474099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.474102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.474297 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.474617 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.474620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.474622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.474719 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.475081 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.475085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.475180 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.475287 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.475415 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:20:44.475582 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.466965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.15.127.189:46377)
0924 07:20:44.467056 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.467060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.467142 (+ 82us) server_negotiation.cc:407] Connection header received
0924 07:20:44.467289 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.467292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.467330 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.467418 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.467815 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.468337 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.469726 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.470386 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.470519 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.472575 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.472579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.472599 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.472626 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.473984 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.474401 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.474406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.474408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.474512 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.474809 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.474814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.474815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.474973 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.475078 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.475345 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.475446 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:20:44.476293 5725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:44.476907 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.476691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46960)
0924 07:20:44.476801 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.476855 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 07:20:44.477610 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.477381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46962)
0924 07:20:44.477519 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.477566 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 07:20:44.477742 5725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:34377: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
I20260924 07:20:44.508893 5725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } attempt: 1
I20260924 07:20:44.509794 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.509472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46968)
0924 07:20:44.509632 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.509700 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:20:44.510589 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.510376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46982)
0924 07:20:44.510481 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.510525 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:20:44.510766 5725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:34377: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
I20260924 07:20:44.512439 5800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } attempt: 1
I20260924 07:20:44.513191 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.512868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:46994)
0924 07:20:44.513051 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.513113 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 07:20:44.513962 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.513725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47010)
0924 07:20:44.513845 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.513900 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 07:20:44.514142 5800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:34377: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
W20260924 07:20:44.576330 5806 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:20:44.576647 5806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:44.576705 5806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:44.576777 5806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:44.580991 5806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:44.581058 5806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:44.581079 5806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:44.581095 5806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:44.581135 5806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:44.582312 5800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } attempt: 2
I20260924 07:20:44.583120 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.582785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47016)
0924 07:20:44.582956 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.583032 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 07:20:44.583894 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.583671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47024)
0924 07:20:44.583785 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.583836 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 07:20:44.584059 5800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:34377: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
I20260924 07:20:44.585902 5725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } attempt: 2
I20260924 07:20:44.586576 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.586327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47040)
0924 07:20:44.586452 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.586514 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 07:20:44.587193 5806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:41409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:36611,127.15.127.189:46377,127.15.127.188:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab
--principal=kudu/127.15.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:44.587291 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.587088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47042)
0924 07:20:44.587194 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.587243 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 07:20:44.587450 5725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:34377: Network error: Client connection negotiation failed: client connection to 127.15.127.188:34377: connect: Connection refused (error 111)
I20260924 07:20:44.588351 5806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:44.589419 5806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:20:44.598090 5806 server_base.cc:1034] running on GCE node
W20260924 07:20:44.598272 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.598295 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:44.598291 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:44.599040 5806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:44.600123 5806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:44.601276 5806 hybrid_clock.cc:648] HybridClock initialized: now 1790234444601264 us; error 24 us; skew 500 ppm
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:44.604257 5806 init.cc:377] Logged in from keytab as kudu/127.15.127.188@KRBTEST.COM (short username kudu)
I20260924 07:20:44.605219 5806 webserver.cc:460] Webserver started at http://127.15.127.188:33965/ using document root <none> and password file <none>
I20260924 07:20:44.605804 5806 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:44.605906 5806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:44.606148 5806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:44.608314 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data/instance:
uuid: "710d8c07872047658e353712da2ce79c"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "b218d86e7a143bdb1a9ce9495d7b9449"
server_key_iv: "7cf8705cee04766de3d74ffb7dfe4188"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.608952 5806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal/instance:
uuid: "710d8c07872047658e353712da2ce79c"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "b218d86e7a143bdb1a9ce9495d7b9449"
server_key_iv: "7cf8705cee04766de3d74ffb7dfe4188"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.612738 5806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 07:20:44.615213 5822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:44.616403 5806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:44.616568 5806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal
uuid: "710d8c07872047658e353712da2ce79c"
format_stamp: "Formatted at 2026-09-24 07:20:44 on dist-test-slave-zx46"
server_key: "b218d86e7a143bdb1a9ce9495d7b9449"
server_key_iv: "7cf8705cee04766de3d74ffb7dfe4188"
server_key_version: "encryptionkey@0"
I20260924 07:20:44.616693 5806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:44.637881 5806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:44.641234 5806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:44.641386 5806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:44.647982 5806 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.188:34377
I20260924 07:20:44.648303 5874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.188:34377 every 8 connection(s)
I20260924 07:20:44.648988 5806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
I20260924 07:20:44.651732 5875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:44.655460 5875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:44.658152 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 5806
I20260924 07:20:44.658285 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790234163642002-15870-0/minicluster-data/master-2/wal/instance
I20260924 07:20:44.658710 15870 external_mini_cluster.cc:1428] Setting key 9832f244503e11f130b6c3637751be63
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:20:44.669902 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.657420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.15.127.190:36611)
0924 07:20:44.657573 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.657578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.658470 (+ 892us) server_negotiation.cc:407] Connection header received
0924 07:20:44.659706 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.659711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.659766 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.659826 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.661269 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.661759 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.662860 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.663151 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.663229 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.665980 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.665986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.665989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.666016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.667686 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.668333 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.668336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.668337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.668414 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.668729 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.668733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.668737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.668992 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.669108 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.669601 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.669732 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 07:20:44.670729 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.657776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:36611 (local address 127.0.0.1:33118)
0924 07:20:44.658295 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.658354 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.659438 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.660101 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.660110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.660574 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.661117 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.661137 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.661873 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.661877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.662735 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.662744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.663286 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.663289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.663440 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.664123 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.664138 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.665774 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.667793 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.667800 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.667809 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.668239 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.668498 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.668500 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.668501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.668622 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.669105 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.669110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.669363 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.669561 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.669863 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":3,"thread_start_us":134,"threads_started":1}
I20260924 07:20:44.670734 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.659441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.15.127.188:34377)
0924 07:20:44.659790 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.659803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.659841 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:20:44.660073 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.660079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.660273 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.660448 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.661137 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.662120 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.663025 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.663468 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.663569 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.666517 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.666522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.666531 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.666551 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.668239 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.668816 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.668820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.668821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.668885 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.669211 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.669216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.669218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.669439 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.669655 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.669898 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.670068 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":3840,"thread_start_us":167,"threads_started":1}
I20260924 07:20:44.672011 5875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:44.675212 5800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } attempt: 3
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:20:44.681996 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.672487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:46377 (local address 127.0.0.1:47174)
0924 07:20:44.672629 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.672643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.672718 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.673346 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.673349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.673516 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.673646 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.673657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.674626 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.674629 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.675408 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.675417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.676004 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.676008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.676110 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.676538 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.676593 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.678087 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.680545 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.680550 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.680553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.680787 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.681096 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.681099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.681101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.681189 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.681530 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.681533 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.681615 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.681731 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.681848 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 07:20:44.682279 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.672803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47174 (local address 127.15.127.189:46377)
0924 07:20:44.672993 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.673018 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.673032 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:20:44.673080 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.673086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.673136 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.673214 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.673782 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.674519 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.675547 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.675883 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.675990 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.678246 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.678250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.678253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.678279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.680419 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.680890 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.680893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.680895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.680980 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.681263 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.681266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.681267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.681428 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.681539 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.681795 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.681896 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 07:20:44.682622 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.672985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33134 (local address 127.15.127.190:36611)
0924 07:20:44.673111 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.673114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.673125 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:44.673206 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.673208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.673240 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.673298 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.674013 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.674696 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.675744 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.676016 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.676108 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.678850 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.678856 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.678861 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.678886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.680590 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.681093 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.681096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.681098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.681186 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.681495 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.681501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.681506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.681716 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.681933 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.682305 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.682426 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:20:44.682893 5875 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:44.685461 5803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.675643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47060)
0924 07:20:44.675793 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.675804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.675888 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.676674 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.676677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.676817 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.676887 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.676891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.677962 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.677965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.678777 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.678791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.679404 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.679415 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.679497 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.679924 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.679940 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.681469 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.683510 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.683518 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.683520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.683766 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.684390 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.684393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.684395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.684486 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.684916 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.684921 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.685053 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.685171 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.685316 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 07:20:44.685544 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.676012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.15.127.188:34377)
0924 07:20:44.676118 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.676121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.676130 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:20:44.676154 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.676156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.676197 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.676294 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.677044 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.677846 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.678890 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.679274 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.679400 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.681606 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.681609 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.681611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.681631 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.683386 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.684176 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.684182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.684187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.684275 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.684593 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.684596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.684598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.684763 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.684916 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.685276 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.685401 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:20:44.686612 5725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } attempt: 3
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:20:44.689500 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: Bootstrap starting.
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
Sep 24 07:20:44 dist-test-slave-zx46 krb5kdc[5644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234444, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:20:44.692618 5800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:44.693624 5800 log.cc:824] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:44.694514 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.684087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.15.127.189:46377)
0924 07:20:44.684189 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.684192 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.684206 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:20:44.684310 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.684315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.684353 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.684418 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.685245 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.685955 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.686937 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.687251 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.687326 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.690486 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.690490 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.690493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.690517 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.692601 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.693124 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.693127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.693128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.693223 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.693523 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.693526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.693528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.693730 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.693903 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.694261 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.694365 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:20:44.694674 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.683706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.15.127.188:34377)
0924 07:20:44.684030 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.684040 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.684052 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:44.684230 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.684233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.684266 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.684338 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.684849 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.685955 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.686925 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.687266 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.687357 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.690173 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.690176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.690178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.690198 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.692601 (+ 2403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.693122 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.693125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.693126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.693355 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.693653 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.693656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.693658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.693950 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.694038 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.694348 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.694467 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":107,"threads_started":1}
I20260924 07:20:44.694909 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.683473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47076)
0924 07:20:44.683931 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.683946 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.684007 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.684435 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.684438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.684616 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.684719 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.684728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.686070 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.686076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.686793 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.686803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.687457 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.687460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.687524 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.688159 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.688190 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.690031 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.692742 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.692745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.692746 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.692970 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.693451 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.693457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.693460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.693563 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.694071 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.694074 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.694153 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.694265 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.694370 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260924 07:20:44.696080 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.686943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:34377 (local address 127.0.0.1:47084)
0924 07:20:44.687138 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:44.687150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:44.687216 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:44.687704 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:44.687707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:44.687852 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:44.687974 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.687984 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.689230 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.689235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.689865 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:44.689874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.690490 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.690493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:44.690583 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.691030 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:44.691040 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:44.692474 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:44.694481 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.694484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.694487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.694722 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.695066 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:44.695071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:44.695074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:44.695167 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:44.695624 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:44.695627 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:44.695721 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 07:20:44.695804 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:20:44.695940 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260924 07:20:44.696656 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.687387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.15.127.188:34377)
0924 07:20:44.687497 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.687500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.687511 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:44.687540 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.687543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.687576 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.688089 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.688205 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.689068 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.689979 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.690345 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.690768 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.692658 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.692664 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.692669 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.692694 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.694346 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.694849 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.694852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.694853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.694930 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.695280 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.695288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.695289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.695512 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.696322 (+ 810us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.696392 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.696516 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:20:44.697788 5800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: No bootstrap required, opened a new log
I20260924 07:20:44.697818 5875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: Bootstrap starting.
I20260924 07:20:44.697877 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:44.699276 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: Bootstrap starting.
I20260924 07:20:44.699115 5699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a35eade882e64bd9a86598973d155250"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33134
I20260924 07:20:44.700321 5875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:44.701195 5875 log.cc:824] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:44.701735 5725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:44.702013 5800 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
I20260924 07:20:44.702260 5800 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b23eee5a194834ae2b0724c5890b7b, State: Initialized, Role: FOLLOWER
I20260924 07:20:44.702551 5725 log.cc:824] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:44.702590 5800 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
I20260924 07:20:44.703150 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.703348 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.703985 5800 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:44.704173 5875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: No bootstrap required, opened a new log
I20260924 07:20:44.704504 5725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: No bootstrap required, opened a new log
I20260924 07:20:44.707361 5725 raft_consensus.cc:348] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
I20260924 07:20:44.707582 5725 raft_consensus.cc:729] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a35eade882e64bd9a86598973d155250, State: Initialized, Role: FOLLOWER
I20260924 07:20:44.707901 5725 consensus_queue.cc:260] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
I20260924 07:20:44.708586 5875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
W20260924 07:20:44.708823 5895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 07:20:44.708843 5875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710d8c07872047658e353712da2ce79c, State: Initialized, Role: FOLLOWER
W20260924 07:20:44.708909 5895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:44.709175 5725 sys_catalog.cc:564] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:44.709275 5875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
I20260924 07:20:44.710604 5875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:44.707589 5732 master_runner.cc:186] Error getting master registration for 127.15.127.190:36611: OK, instance_id { permanent_uuid: "a35eade882e64bd9a86598973d155250" instance_seqno: 1790234444224695 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:20:44.711407 5732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:44.711572 5699 raft_consensus.cc:482] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:20:44.711684 5699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:44.712966 5699 raft_consensus.cc:504] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
I20260924 07:20:44.713394 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.713605 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.713932 5699 leader_election.cc:288] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [CANDIDATE]: Term 1 election: Requested vote from peers c6b23eee5a194834ae2b0724c5890b7b (127.15.127.189:46377), 710d8c07872047658e353712da2ce79c (127.15.127.188:34377)
I20260924 07:20:44.714818 5774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a35eade882e64bd9a86598973d155250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c6b23eee5a194834ae2b0724c5890b7b"
I20260924 07:20:44.715096 5774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:44.716367 5774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a35eade882e64bd9a86598973d155250 in term 1.
I20260924 07:20:44.714394 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.715629 5849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a35eade882e64bd9a86598973d155250" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710d8c07872047658e353712da2ce79c"
I20260924 07:20:44.716897 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.716964 5849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [term 0 FOLLOWER]: Advancing to term 1
W20260924 07:20:44.718114 5909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:44.718194 5909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:44.718417 5849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a35eade882e64bd9a86598973d155250 in term 1.
I20260924 07:20:44.718902 5673 leader_election.cc:302] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a35eade882e64bd9a86598973d155250, c6b23eee5a194834ae2b0724c5890b7b; no voters:
I20260924 07:20:44.719398 5897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:44.719619 5897 raft_consensus.cc:686] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [term 1 LEADER]: Becoming Leader. State: Replica: a35eade882e64bd9a86598973d155250, State: Running, Role: LEADER
I20260924 07:20:44.719969 5897 consensus_queue.cc:237] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } }
W20260924 07:20:44.721637 5678 tablet.cc:2369] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:44.724893 5657 master_runner.cc:263] No leader master found from master a35eade882e64bd9a86598973d155250
I20260924 07:20:44.724967 5657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:20:44.727211 5920 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:44.727293 5920 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:44.727623 5929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a35eade882e64bd9a86598973d155250. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.727749 5929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:44.730891 5932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:44.731853 5932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:44.733315 5729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.717715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.15.127.190:36611)
0924 07:20:44.717816 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.717820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.718947 (+ 1127us) server_negotiation.cc:407] Connection header received
0924 07:20:44.719192 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.719195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.719242 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.719298 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.720502 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.721200 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.727462 (+ 6262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.727867 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.727966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.729751 (+ 1785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.729754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.729757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.729782 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.731466 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.731965 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.731969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.731971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.732052 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.732336 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.732340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.732342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.732587 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.732691 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.733050 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.733172 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:20:44.735731 5879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.716713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.15.127.188:34377)
0924 07:20:44.716828 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.716833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.718874 (+ 2041us) server_negotiation.cc:407] Connection header received
0924 07:20:44.719065 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.719068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.719115 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.719166 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.720121 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.720922 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.722817 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.723134 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.723211 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.726161 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.726165 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.726168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.726206 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.728768 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.734365 (+ 5597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.734369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.734371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.734481 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.734790 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.734794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.734796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.735012 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.735119 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.735451 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.735575 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:20:44.741803 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:44.718458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.15.127.189:46377)
0924 07:20:44.718575 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.718579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.729276 (+ 10697us) server_negotiation.cc:407] Connection header received
0924 07:20:44.729829 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.729833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.729880 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.729930 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.733107 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.733892 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.735637 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:44.735976 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:44.736051 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:44.737855 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:44.737859 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:44.737862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:44.737891 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:44.739669 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.740150 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.740153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.740155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.740239 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:44.740561 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:44.740565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:44.740567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:44.740765 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:44.740891 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:44.741499 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 07:20:44.741615 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:20:44.742337 5849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [term 1 FOLLOWER]: Refusing update from remote peer a35eade882e64bd9a86598973d155250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:44.743218 5929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:44.743625 5774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [term 1 FOLLOWER]: Refusing update from remote peer a35eade882e64bd9a86598973d155250: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:44.747466 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a35eade882e64bd9a86598973d155250. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.747589 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.747961 5929 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:44.751075 5929 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.751184 5929 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:44.751336 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a35eade882e64bd9a86598973d155250. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.751441 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.752723 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.752786 5933 mvcc.cc:205] Tried to move back new op lower bound from 7332800285656272896 to 7332800285576450048. Current Snapshot: MvccSnapshot[applied={T|T < 7332800285656272896}]
I20260924 07:20:44.752836 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.753082 5934 mvcc.cc:205] Tried to move back new op lower bound from 7332800285656272896 to 7332800285576450048. Current Snapshot: MvccSnapshot[applied={T|T < 7332800285656272896}]
I20260924 07:20:44.754514 5932 catalog_manager.cc:1347] Generated new cluster ID: f8253b4ccf434788af35a5b78082b264
I20260924 07:20:44.754577 5932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:44.759821 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.759958 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:44.760946 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a35eade882e64bd9a86598973d155250" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a35eade882e64bd9a86598973d155250" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 36611 } } peers { permanent_uuid: "c6b23eee5a194834ae2b0724c5890b7b" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 46377 } } peers { permanent_uuid: "710d8c07872047658e353712da2ce79c" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 34377 } } }
I20260924 07:20:44.761080 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:44.761633 5936 mvcc.cc:205] Tried to move back new op lower bound from 7332800285656272896 to 7332800285576450048. Current Snapshot: MvccSnapshot[applied={T|T < 7332800285656272896}]
I20260924 07:20:44.781396 5932 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:44.782639 5932 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:44.794555 5932 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a35eade882e64bd9a86598973d155250: Generated new TSK 0
I20260924 07:20:44.795361 5932 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:20:44.832322 5879 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.829431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47116 (local address 127.15.127.188:34377)
0924 07:20:44.829577 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.829582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.831212 (+ 1630us) server_negotiation.cc:407] Connection header received
0924 07:20:44.831452 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.831456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.831510 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.831564 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.832183 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47116: BlockingRecv error: recv got EOF from 127.0.0.1:47116 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 07:20:44.833736 5729 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.830010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.15.127.190:36611)
0924 07:20:44.830124 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.830128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.832847 (+ 2719us) server_negotiation.cc:407] Connection header received
0924 07:20:44.833036 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.833039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.833089 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.833140 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.833621 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33152: BlockingRecv error: recv got EOF from 127.0.0.1:33152 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 07:20:44.835163 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.830721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.15.127.189:46377)
0924 07:20:44.830852 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.830858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.834208 (+ 3350us) server_negotiation.cc:407] Connection header received
0924 07:20:44.834451 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.834455 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.834511 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.834563 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.835043 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47198: BlockingRecv error: recv got EOF from 127.0.0.1:47198 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 07:20:44.839429 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.837440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47206 (local address 127.15.127.189:46377)
0924 07:20:44.837534 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.837538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.838544 (+ 1006us) server_negotiation.cc:407] Connection header received
0924 07:20:44.838770 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.838773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.838846 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.838894 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.839328 (+ 434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47206: BlockingRecv error: recv got EOF from 127.0.0.1:47206 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 07:20:44.844085 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.842762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.15.127.189:46377)
0924 07:20:44.842849 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.842853 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.843263 (+ 410us) server_negotiation.cc:407] Connection header received
0924 07:20:44.843439 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.843442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.843486 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.843532 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.844003 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47212: BlockingRecv error: recv got EOF from 127.0.0.1:47212 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 07:20:44.849344 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.848023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.15.127.189:46377)
0924 07:20:44.848109 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.848113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.848525 (+ 412us) server_negotiation.cc:407] Connection header received
0924 07:20:44.848682 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.848685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.848725 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.848768 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.849249 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47228: BlockingRecv error: recv got EOF from 127.0.0.1:47228 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 07:20:44.856961 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.855568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.15.127.189:46377)
0924 07:20:44.855691 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.855696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.856163 (+ 467us) server_negotiation.cc:407] Connection header received
0924 07:20:44.856342 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.856345 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.856388 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.856434 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.856878 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47244: BlockingRecv error: recv got EOF from 127.0.0.1:47244 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 07:20:44.868660 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.867172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47258 (local address 127.15.127.189:46377)
0924 07:20:44.867281 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.867286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.867783 (+ 497us) server_negotiation.cc:407] Connection header received
0924 07:20:44.867967 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.867970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.868017 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.868065 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.868565 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47258: BlockingRecv error: recv got EOF from 127.0.0.1:47258 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:20:44.889201 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.887383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.15.127.189:46377)
0924 07:20:44.887527 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.887532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.888086 (+ 554us) server_negotiation.cc:407] Connection header received
0924 07:20:44.888322 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.888326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.888384 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.888442 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.889081 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47262: BlockingRecv error: recv got EOF from 127.0.0.1:47262 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 07:20:44.925523 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.923794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.15.127.189:46377)
0924 07:20:44.923915 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.923919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.924459 (+ 540us) server_negotiation.cc:407] Connection header received
0924 07:20:44.924668 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.924672 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.924726 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.924790 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.925410 (+ 620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47278: BlockingRecv error: recv got EOF from 127.0.0.1:47278 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:20:44.993194 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:44.992091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.15.127.189:46377)
0924 07:20:44.992260 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:44.992264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:44.992337 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:20:44.992460 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:44.992463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:44.992511 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:44.992601 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:44.993076 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47294: BlockingRecv error: recv got EOF from 127.0.0.1:47294 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 07:20:45.124226 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:45.123115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.15.127.189:46377)
0924 07:20:45.123290 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.123294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.123393 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:20:45.123518 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.123521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.123572 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.123645 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:45.124102 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47304: BlockingRecv error: recv got EOF from 127.0.0.1:47304 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 07:20:45.383663 5805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:45.382536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.15.127.189:46377)
0924 07:20:45.382673 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.382678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.382780 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:20:45.382952 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.382956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.383015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.383108 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:45.383533 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47312: BlockingRecv error: recv got EOF from 127.0.0.1:47312 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 07:20:45.710366 5895 catalog_manager.cc:1259] Loaded cluster ID: f8253b4ccf434788af35a5b78082b264
I20260924 07:20:45.710489 5895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: loading cluster ID for follower catalog manager: success
I20260924 07:20:45.712343 5895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: acquiring CA information for follower catalog manager: success
I20260924 07:20:45.713562 5895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c6b23eee5a194834ae2b0724c5890b7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:20:45.728449 5920 catalog_manager.cc:1259] Loaded cluster ID: f8253b4ccf434788af35a5b78082b264
I20260924 07:20:45.728515 5920 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: loading cluster ID for follower catalog manager: success
I20260924 07:20:45.730086 5920 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: acquiring CA information for follower catalog manager: success
I20260924 07:20:45.731158 5920 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 710d8c07872047658e353712da2ce79c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:20:45.901683 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.897957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.15.127.189:46377)
0924 07:20:45.898250 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.898254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.898270 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:20:45.898416 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.898418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.898544 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.898638 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:45.898876 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.899352 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.900166 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.900373 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.900445 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.900716 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:45.901224 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:45.901316 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.901422 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.901504 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":89,"threads_started":1}
I20260924 07:20:45.908430 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.904942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58118 (local address 127.15.127.188:34377)
0924 07:20:45.905229 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.905233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.905245 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:45.905316 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.905319 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.905494 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.905603 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:45.905765 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.906356 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.907140 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.907350 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.907421 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.907673 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:45.908077 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:45.908162 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.908231 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.908275 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":109,"threads_started":1}
I20260924 07:20:45.916239 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.912112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.15.127.188:34377)
0924 07:20:45.912217 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.912220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.912466 (+ 246us) server_negotiation.cc:407] Connection header received
0924 07:20:45.912673 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.912677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.912723 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.912770 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:45.913219 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.913707 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.915136 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.915344 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.915480 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.915740 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:45.915836 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:45.915904 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.916049 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.916107 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:20:45.916463 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.911890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.15.127.189:46377)
0924 07:20:45.912037 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.912041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.912408 (+ 367us) server_negotiation.cc:407] Connection header received
0924 07:20:45.912623 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.912626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.912676 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.912747 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:45.913026 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.913817 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.915099 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.915427 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.915541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.915918 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:45.915991 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:45.916062 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.916245 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.916304 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 07:20:45.916756 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:45.911817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.15.127.190:36611)
0924 07:20:45.912048 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:45.912053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:45.912475 (+ 422us) server_negotiation.cc:407] Connection header received
0924 07:20:45.912612 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:45.912615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:45.912776 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:45.912886 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:45.913235 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.914015 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.914849 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:45.915174 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:45.915262 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:45.915665 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:45.916049 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:45.916473 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:45.916512 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 07:20:45.916587 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":65,"threads_started":1}
I20260924 07:20:45.927290 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5657
I20260924 07:20:45.933784 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5732
I20260924 07:20:45.939407 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 5806
2026-09-24T07:20:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3028 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:20:45 dist-test-slave-zx46 krb5kdc[6024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:20:45 dist-test-slave-zx46 krb5kdc[6024](info): set up 2 sockets
Sep 24 07:20:45 dist-test-slave-zx46 krb5kdc[6024](info): commencing operation
krb5kdc: starting...
W20260924 07:20:46.969373 15870 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/test-user.keytab.
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:20:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:20:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.127.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
Entry for principal kudu/127.15.127.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab.
I20260924 07:20:47.088436 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33437
--webserver_interface=127.15.127.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33437,127.15.127.189:38255,127.15.127.188:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:20:47.194140 6037 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:20:47.194453 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:47.194527 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:47.194612 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:47.198490 6037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:47.198570 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:47.198606 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:47.198639 6037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:47.198666 6037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:47.203487 6037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33437,127.15.127.189:38255,127.15.127.188:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.190.keytab
--principal=kudu/127.15.127.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.190:33437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.127.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:47.204522 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:47.205535 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:47.213342 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:47.213384 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:47.213388 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:47.213469 6037 server_base.cc:1034] running on GCE node
I20260924 07:20:47.214149 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:47.215212 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:47.216409 6037 hybrid_clock.cc:648] HybridClock initialized: now 1790234447216392 us; error 33 us; skew 500 ppm
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:47.219154 6037 init.cc:377] Logged in from keytab as kudu/127.15.127.190@KRBTEST.COM (short username kudu)
I20260924 07:20:47.220022 6037 webserver.cc:460] Webserver started at http://127.15.127.190:36101/ using document root <none> and password file <none>
I20260924 07:20:47.220562 6037 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:47.220647 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:47.220867 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:47.222854 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data/instance:
uuid: "ccaa7f752f254c5b9200fb381663e011"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "61518587e3b266dc968dd3c1a1f4e7f5"
server_key_iv: "ba0ce76700cfe520b23f2344c606453a"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.223409 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal/instance:
uuid: "ccaa7f752f254c5b9200fb381663e011"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "61518587e3b266dc968dd3c1a1f4e7f5"
server_key_iv: "ba0ce76700cfe520b23f2344c606453a"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.227099 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 07:20:47.229452 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:47.230556 6037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:47.230707 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal
uuid: "ccaa7f752f254c5b9200fb381663e011"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "61518587e3b266dc968dd3c1a1f4e7f5"
server_key_iv: "ba0ce76700cfe520b23f2344c606453a"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.230815 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:47.256984 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:47.260509 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:47.260749 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:47.267678 6037 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.190:33437
I20260924 07:20:47.267946 6104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.190:33437 every 8 connection(s)
I20260924 07:20:47.268692 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/data/info.pb
I20260924 07:20:47.271487 6105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:47.274865 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 6037
I20260924 07:20:47.274994 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-0/wal/instance
I20260924 07:20:47.275305 15870 external_mini_cluster.cc:1428] Setting key 4b7bafadc9984cf6bca7f9eb8bdecddf
I20260924 07:20:47.275420 6105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.127.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.189@KRBTEST.COM" created.
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:47.290971 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.278067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.15.127.190:33437)
0924 07:20:47.278484 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.278504 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.278760 (+ 256us) server_negotiation.cc:407] Connection header received
0924 07:20:47.279817 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.279821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.280004 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.280192 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.280991 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.281872 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.282802 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.283210 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.283436 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.286299 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.286308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.286325 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.286356 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.288017 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.288585 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.288589 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.288590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.288687 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.289098 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.289102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.289105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.289475 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.289575 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.290042 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.290235 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":260,"thread_start_us":131,"threads_started":1}
I20260924 07:20:47.290977 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.278068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33437 (local address 127.0.0.1:58142)
0924 07:20:47.278622 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.278657 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.279491 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.280187 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.280191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.280622 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.280844 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.280856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.282002 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.282010 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.282687 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.282695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.283357 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.283361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.283525 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.284102 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.284117 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.286145 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.288129 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.288141 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.288153 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.288478 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.288789 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.288795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.288800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.288950 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.289601 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.289611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.289784 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.289989 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.290319 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":85,"spinlock_wait_cycles":13056,"thread_start_us":154,"threads_started":1}
I20260924 07:20:47.293251 6105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:47.293929 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.293679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51492)
0924 07:20:47.293809 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.293874 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 07:20:47.294118 6053 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
I20260924 07:20:47.295027 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.294705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51504)
0924 07:20:47.294882 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.294941 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 07:20:47.295192 6105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:38255: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab.
Entry for principal kudu/127.15.127.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab.
I20260924 07:20:47.300294 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:38255
--webserver_interface=127.15.127.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab
--principal=kudu/127.15.127.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33437,127.15.127.189:38255,127.15.127.188:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:47.344425 6105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } attempt: 1
I20260924 07:20:47.345479 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.345142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51506)
0924 07:20:47.345310 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.345385 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 07:20:47.346187 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.345973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51520)
0924 07:20:47.346076 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.346126 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:20:47.346328 6105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:38255: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
W20260924 07:20:47.407858 6112 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:20:47.408106 6112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:47.408128 6112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:47.408164 6112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:47.412168 6112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:47.412220 6112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:47.412237 6112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:47.412271 6112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:47.412286 6112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:47.414539 6105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } attempt: 2
I20260924 07:20:47.415349 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.415008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51524)
0924 07:20:47.415191 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.415261 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:20:47.416101 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.415888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51536)
0924 07:20:47.416005 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.416050 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:20:47.416299 6105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.189:38255: Network error: Client connection negotiation failed: client connection to 127.15.127.189:38255: connect: Connection refused (error 111)
I20260924 07:20:47.417968 6112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33437,127.15.127.189:38255,127.15.127.188:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.189.keytab
--principal=kudu/127.15.127.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.189:38255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.127.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:47.418969 6112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:47.419951 6112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:20:47.428120 6118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:47.428186 6117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:47.428113 6120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:47.428462 6112 server_base.cc:1034] running on GCE node
I20260924 07:20:47.428988 6112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:47.430198 6112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:47.431376 6112 hybrid_clock.cc:648] HybridClock initialized: now 1790234447431362 us; error 26 us; skew 500 ppm
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:47.434095 6112 init.cc:377] Logged in from keytab as kudu/127.15.127.189@KRBTEST.COM (short username kudu)
I20260924 07:20:47.435009 6112 webserver.cc:460] Webserver started at http://127.15.127.189:33585/ using document root <none> and password file <none>
I20260924 07:20:47.435557 6112 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:47.435647 6112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:47.435868 6112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:47.437873 6112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data/instance:
uuid: "386f9e2791834b66816bf885644a20e0"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "1d9c50990512b979a8f51e91a18f26d2"
server_key_iv: "74493f7890dad7eefc18a0f2d504cea5"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.438421 6112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal/instance:
uuid: "386f9e2791834b66816bf885644a20e0"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "1d9c50990512b979a8f51e91a18f26d2"
server_key_iv: "74493f7890dad7eefc18a0f2d504cea5"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.442018 6112 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:47.444351 6127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:47.445549 6112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:20:47.445686 6112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal
uuid: "386f9e2791834b66816bf885644a20e0"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "1d9c50990512b979a8f51e91a18f26d2"
server_key_iv: "74493f7890dad7eefc18a0f2d504cea5"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.445793 6112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:47.463048 6112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:47.465873 6112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:47.466071 6112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:47.472779 6112 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.189:38255
I20260924 07:20:47.473084 6179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.189:38255 every 8 connection(s)
I20260924 07:20:47.473800 6112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/data/info.pb
I20260924 07:20:47.476205 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 6112
I20260924 07:20:47.476365 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-1/wal/instance
I20260924 07:20:47.476666 15870 external_mini_cluster.cc:1428] Setting key 37b67ab32f38935382df34bb8ba50cf8
I20260924 07:20:47.476861 6180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:47.480666 6180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.127.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.127.188@KRBTEST.COM" created.
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:47.495208 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.482616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58144 (local address 127.15.127.190:33437)
0924 07:20:47.482749 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.482753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.483756 (+ 1003us) server_negotiation.cc:407] Connection header received
0924 07:20:47.484742 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.484746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.484809 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.484918 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.486181 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.486744 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.487948 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.488373 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.488517 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.491177 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.491184 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.491190 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.491223 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.492966 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.493496 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.493503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.493508 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.493607 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.493940 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.493953 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.493957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.494236 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.494350 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.494855 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.495011 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:20:47.495985 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.483116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33437 (local address 127.0.0.1:58144)
0924 07:20:47.483603 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.483644 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.484536 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.485110 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.485117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.485514 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.485998 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.486015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.486874 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.486876 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.487816 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.487826 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.488532 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.488535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.488673 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.489318 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.489343 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.491004 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.493117 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.493122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.493134 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.493384 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.493712 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.493714 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.493716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.493846 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.494341 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.494346 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.494595 (+ 249us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.494753 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.495003 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":130,"threads_started":1}
I20260924 07:20:47.497550 6180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.127.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab.
Entry for principal kudu/127.15.127.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab.
I20260924 07:20:47.504902 15870 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
/tmp/dist-test-taskPYFHG9/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:44581
--webserver_interface=127.15.127.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab
--principal=kudu/127.15.127.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.127.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.127.190:33437,127.15.127.189:38255,127.15.127.188:44581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:20:47.507009 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.498086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51550)
0924 07:20:47.498185 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.498200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.498275 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.498979 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.498982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.499154 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.499242 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.499247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.500320 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.500324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.500945 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.500967 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.501552 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.501555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.501664 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.502082 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.502092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.503244 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.505399 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.505405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.505411 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.505674 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.505956 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.505962 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.505967 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.506066 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.506632 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.506634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.506708 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.506790 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.506886 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:20:47.507318 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.498234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.15.127.189:38255)
0924 07:20:47.498638 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.498648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.498665 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:20:47.498698 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.498701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.498851 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.498955 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.499343 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.500177 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.501097 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.501446 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.501559 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.503361 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.503369 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.503380 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.503400 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.505299 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.505787 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.505791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.505792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.505864 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.506171 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.506177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.506182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.506523 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.506663 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.506911 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.507073 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":231,"threads_started":1}
I20260924 07:20:47.507534 6105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } attempt: 3
I20260924 07:20:47.509346 6180 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:47.510088 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.509762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58764)
0924 07:20:47.509901 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.510001 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 07:20:47.510279 6131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
I20260924 07:20:47.512101 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.511860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58766)
0924 07:20:47.511971 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.512018 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:20:47.512247 6180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:44581: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
I20260924 07:20:47.516762 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.507964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51562)
0924 07:20:47.508171 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.508186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.508264 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.508624 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.508630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.508827 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.509506 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.509520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.510409 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.510413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.510969 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.510977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.511484 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.511487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.511588 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.512345 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.512378 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.513651 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.515280 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.515284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.515286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.515498 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.515789 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.515793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.515795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.515877 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.516256 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.516260 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.516361 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.516469 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.516607 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 07:20:47.516760 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.508007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.15.127.189:38255)
0924 07:20:47.508401 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.508405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.508417 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:47.508445 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.508448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.508496 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.508567 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.509641 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.510278 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.511088 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.511368 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.511485 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.513776 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.513780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.513787 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.513809 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.515172 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.515597 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.515602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.515604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.515685 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.515971 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.515975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.515976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.516160 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.516263 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.516524 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.516626 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260924 07:20:47.517421 6105 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:47.518098 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.517879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58782)
0924 07:20:47.517982 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.518029 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:20:47.518708 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.518488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58796)
0924 07:20:47.518614 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.518656 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 07:20:47.518850 6105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:44581: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
I20260924 07:20:47.550004 6105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } attempt: 1
I20260924 07:20:47.550882 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.550531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58798)
0924 07:20:47.550711 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.550786 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 07:20:47.551621 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.551398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58808)
0924 07:20:47.551505 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.551556 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 07:20:47.551780 6105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:44581: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
I20260924 07:20:47.561519 6180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } attempt: 1
I20260924 07:20:47.562340 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.562081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58824)
0924 07:20:47.562204 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.562261 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 07:20:47.563045 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.562822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58834)
0924 07:20:47.562931 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.562976 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 07:20:47.563208 6180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:44581: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
W20260924 07:20:47.619134 6186 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:20:47.619405 6186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:20:47.619426 6186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:20:47.619462 6186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:20:47.623453 6186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:20:47.623517 6186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:20:47.623535 6186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:20:47.623553 6186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:20:47.623567 6186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:20:47.626983 6105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } attempt: 2
I20260924 07:20:47.627852 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.627489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58840)
0924 07:20:47.627687 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.627758 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 07:20:47.628556 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.628354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58854)
0924 07:20:47.628453 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.628500 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:20:47.628700 6105 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:44581: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
I20260924 07:20:47.629451 6186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.127.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.127.190:33437,127.15.127.189:38255,127.15.127.188:44581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/krb5kdc/kudu_127.15.127.188.keytab
--principal=kudu/127.15.127.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.127.188:44581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.127.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 06:29:36 UTC on ccycloud.quasar-hawwsb.root.comops.site
build id 4669403
I20260924 07:20:47.630510 6186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:20:47.631390 6180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } attempt: 2
I20260924 07:20:47.631623 6186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:20:47.632079 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.631833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58858)
0924 07:20:47.631951 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.632018 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:20:47.632783 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.632570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58862)
0924 07:20:47.632679 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.632731 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 07:20:47.632906 6180 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.127.188:44581: Network error: Client connection negotiation failed: client connection to 127.15.127.188:44581: connect: Connection refused (error 111)
W20260924 07:20:47.640234 6193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:47.640280 6186 server_base.cc:1034] running on GCE node
W20260924 07:20:47.640241 6192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:20:47.640241 6195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:20:47.641101 6186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:20:47.642138 6186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:20:47.643292 6186 hybrid_clock.cc:648] HybridClock initialized: now 1790234447643262 us; error 42 us; skew 500 ppm
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:20:47.646061 6186 init.cc:377] Logged in from keytab as kudu/127.15.127.188@KRBTEST.COM (short username kudu)
I20260924 07:20:47.646889 6186 webserver.cc:460] Webserver started at http://127.15.127.188:44305/ using document root <none> and password file <none>
I20260924 07:20:47.647395 6186 fs_manager.cc:359] Metadata directory not provided
I20260924 07:20:47.647460 6186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:20:47.647620 6186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:20:47.649803 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data/instance:
uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "6a2154313cb2c0cb5d8cbbf6a5dc339c"
server_key_iv: "125d43c1115b5ee3cf3d80e8b1d91ee5"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.650362 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal/instance:
uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "6a2154313cb2c0cb5d8cbbf6a5dc339c"
server_key_iv: "125d43c1115b5ee3cf3d80e8b1d91ee5"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.653957 6186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 07:20:47.656242 6202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:20:47.657514 6186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 07:20:47.657606 6186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal
uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04"
format_stamp: "Formatted at 2026-09-24 07:20:47 on dist-test-slave-zx46"
server_key: "6a2154313cb2c0cb5d8cbbf6a5dc339c"
server_key_iv: "125d43c1115b5ee3cf3d80e8b1d91ee5"
server_key_version: "encryptionkey@0"
I20260924 07:20:47.657684 6186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:20:47.675102 6186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:20:47.679550 6186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:20:47.679737 6186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:20:47.688032 6186 rpc_server.cc:307] RPC server started. Bound to: 127.15.127.188:44581
I20260924 07:20:47.688361 6254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.127.188:44581 every 8 connection(s)
I20260924 07:20:47.689215 6186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/data/info.pb
I20260924 07:20:47.692049 15870 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu as pid 6186
I20260924 07:20:47.692191 15870 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYFHG9/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790234163642002-15870-0/minicluster-data/master-2/wal/instance
I20260924 07:20:47.692539 15870 external_mini_cluster.cc:1428] Setting key 400b7e1b1698eae177a691dc8ff619b6
I20260924 07:20:47.692920 6255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:20:47.698282 6255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:47.709878 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.694375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.15.127.188:44581)
0924 07:20:47.694811 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.694829 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.694875 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:20:47.695550 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.695566 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.695891 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.696159 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.697075 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.698844 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.699986 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.700760 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.701476 (+ 716us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.704143 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.704152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.704166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.704203 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.706688 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.707241 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.707248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.707255 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.707406 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.707748 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.707750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.707751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.708115 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.708334 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.708596 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.708863 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":151,"threads_started":1}
I20260924 07:20:47.710974 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.700148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58148 (local address 127.15.127.190:33437)
0924 07:20:47.700269 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.700274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.700569 (+ 295us) server_negotiation.cc:407] Connection header received
0924 07:20:47.700755 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.700758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.700809 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.700864 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.701853 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.702357 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.703347 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.703766 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.703913 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.706493 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.706496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.706499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.706530 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.708648 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.709291 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.709294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.709296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.709376 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.709726 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.709731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.709734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.710014 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.710118 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.710608 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.710754 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:20:47.711593 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.700119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.190:33437 (local address 127.0.0.1:58148)
0924 07:20:47.700453 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.700500 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.700648 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.701052 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.701058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.701538 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.701723 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.701730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.702476 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.702479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.703209 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.703220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.703959 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.703964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.704084 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.704653 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.704672 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.706347 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.708786 (+ 2439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.708795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.708802 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.709179 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.709509 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.709512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.709514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.709604 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.710197 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.710204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.710345 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.711133 (+ 788us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.711341 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":107,"threads_started":1}
I20260924 07:20:47.712898 6255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.190@KRBTEST.COM
I20260924 07:20:47.723141 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.713638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.189:38255 (local address 127.0.0.1:51568)
0924 07:20:47.714024 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.714036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.714128 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.714437 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.714440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.714595 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.714687 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.714692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.715699 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.715702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.716518 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.716525 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.717220 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.717224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.717352 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.718144 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.718162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.719512 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.721466 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.721472 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.721475 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.721718 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.722058 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.722060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.722061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.722170 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.722640 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.722643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.722732 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.722841 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.722978 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260924 07:20:47.723230 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.713853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51568 (local address 127.15.127.189:38255)
0924 07:20:47.713955 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.713959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.714078 (+ 119us) server_negotiation.cc:407] Connection header received
0924 07:20:47.714217 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.714221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.714259 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.714315 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.714805 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.715579 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.716633 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.716963 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.717045 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.719675 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.719681 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.719688 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.719726 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.721351 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.721840 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.721847 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.721851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.721939 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.722287 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.722291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.722292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.722530 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.722612 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.722913 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.723055 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:20:47.723783 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.713175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.15.127.190:33437)
0924 07:20:47.713268 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.713272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.713283 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:47.713322 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.713325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.713361 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.713434 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.714299 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.715164 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.716460 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.716822 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.716908 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.720154 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.720158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.720160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.720184 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.721601 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.722118 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.722124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.722126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.722199 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.722471 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.722473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.722474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.722703 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.722916 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.723073 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.723182 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:20:47.723883 6255 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:20:47.724035 6180 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } attempt: 3
I20260924 07:20:47.727867 6105 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } attempt: 3
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.189@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.188@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.127.189@KRBTEST.COM
Sep 24 07:20:47 dist-test-slave-zx46 krb5kdc[6024](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234447, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.127.190@KRBTEST.COM for kudu/127.15.127.188@KRBTEST.COM
I20260924 07:20:47.735185 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.724365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58866)
0924 07:20:47.724476 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.724490 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.724561 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.726406 (+ 1845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.726408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.726575 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.726653 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.726659 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.727608 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.727611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.728330 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.728340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.729093 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.729098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.729205 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.729629 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.729647 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.731034 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.733273 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.733278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.733283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.733694 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.733995 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.733999 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.734001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.734082 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.734586 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.734588 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.734761 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.734862 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.735013 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:20:47.735677 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.724370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58878)
0924 07:20:47.724630 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.724644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.724712 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.724969 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.724974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.725332 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.725422 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.725428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.726476 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.726479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.727221 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.727230 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.727797 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.727801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.728401 (+ 600us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.728882 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.728897 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.731468 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.733730 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.733736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.733739 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.734172 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.734564 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.734567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.734569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.734655 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.735120 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.735126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.735225 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.735309 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.735451 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":484}
I20260924 07:20:47.735695 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.724855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.15.127.188:44581)
0924 07:20:47.725721 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.725730 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.725741 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:20:47.726237 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.726240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.726280 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.726372 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.726773 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.727512 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.728451 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.728828 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.728930 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.731221 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.731225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.731228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.731253 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.733164 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.733798 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.733801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.733810 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.733893 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.734178 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.734184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.734186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.734460 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.734551 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.735385 (+ 834us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.735499 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":207,"threads_started":1}
I20260924 07:20:47.736281 6258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.724569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.15.127.188:44581)
0924 07:20:47.724666 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.724670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.724680 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:20:47.724796 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.724799 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.724845 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.724922 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.725551 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.726355 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.727357 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.727693 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.727796 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.731628 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.731630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.731632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.731651 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.733602 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.734293 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.734296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.734298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.734437 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.734774 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.734777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.734779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.735001 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.735971 (+ 970us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.736043 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.736156 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":4992}
I20260924 07:20:47.737058 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.725237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.15.127.189:38255)
0924 07:20:47.725360 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.725363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.725375 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:47.725402 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.725404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.725435 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.725525 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.726091 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.726702 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.727446 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.727747 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.728320 (+ 573us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.732414 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.732421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.732424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.732449 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.734307 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.734851 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.734857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.734858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.734935 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.735603 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.735606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.735608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.736187 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.736386 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.736769 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.736893 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":47}
I20260924 07:20:47.739503 6109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.728203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.127.188:44581 (local address 127.0.0.1:58884)
0924 07:20:47.728521 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 07:20:47.728537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:20:47.728607 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:20:47.729065 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:20:47.729069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:20:47.729227 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:20:47.729319 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.729327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.730470 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.730473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.731274 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:20:47.731282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.732280 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.732285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:20:47.732394 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.733573 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:20:47.733588 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:20:47.735079 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:20:47.737654 (+ 2575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.737657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.737659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.737914 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.738506 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:20:47.738512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:20:47.738517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:20:47.738610 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:20:47.739016 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:20:47.739019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:20:47.739118 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:20:47.739235 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 07:20:47.739363 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260924 07:20:47.739545 6180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: Bootstrap starting.
I20260924 07:20:47.739976 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.728264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.15.127.188:44581)
0924 07:20:47.728806 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.728814 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.728826 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:20:47.728859 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.728861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.728897 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.729434 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.729567 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.730352 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.731391 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.731772 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.731879 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.735207 (+ 3328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.735210 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.735212 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.735232 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.737388 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.738308 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.738311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.738313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.738393 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.738723 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.738726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.738728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.738908 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.739012 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.739716 (+ 704us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.739822 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":113,"threads_started":1}
I20260924 07:20:47.739944 6255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: Bootstrap starting.
I20260924 07:20:47.742139 6180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:47.742578 6255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:47.742801 6180 log.cc:824] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:47.742926 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011: Bootstrap starting.
I20260924 07:20:47.743364 6255 log.cc:824] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:47.744895 6180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: No bootstrap required, opened a new log
I20260924 07:20:47.745344 6105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011: Neither blocks nor log segments found. Creating new log.
I20260924 07:20:47.745400 6255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: No bootstrap required, opened a new log
I20260924 07:20:47.746160 6105 log.cc:824] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011: Log is configured to *not* fsync() on all Append() calls
I20260924 07:20:47.747005 15870 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:20:47.748481 6180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.748694 6180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 386f9e2791834b66816bf885644a20e0, State: Initialized, Role: FOLLOWER
I20260924 07:20:47.749037 6180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.749312 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011: No bootstrap required, opened a new log
I20260924 07:20:47.748324 6079 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ccaa7f752f254c5b9200fb381663e011"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58156
I20260924 07:20:47.750154 6255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.750378 6255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab15eaa1bb4329bdfe9e2d0b0abc04, State: Initialized, Role: FOLLOWER
I20260924 07:20:47.750378 6180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:47.750677 6255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.751885 6255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:47.752178 6105 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.752393 6105 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccaa7f752f254c5b9200fb381663e011, State: Initialized, Role: FOLLOWER
I20260924 07:20:47.752730 6105 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.754791 6105 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:20:47.754885 6079 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:20:47.755004 6079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:47.756233 6079 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.756942 6265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.757133 6265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:20:47.757351 6289 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:20:47.757411 6289 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:47.757874 6266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.757977 6266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:47.758064 6229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccaa7f752f254c5b9200fb381663e011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04"
I20260924 07:20:47.758260 6229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:47.754076 6112 master_runner.cc:186] Error getting master registration for 127.15.127.190:33437: OK, instance_id { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" instance_seqno: 1790234447265816 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:20:47.758430 6112 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:47.757458 6079 leader_election.cc:288] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [CANDIDATE]: Term 1 election: Requested vote from peers 386f9e2791834b66816bf885644a20e0 (127.15.127.189:38255), 0aab15eaa1bb4329bdfe9e2d0b0abc04 (127.15.127.188:44581)
I20260924 07:20:47.759601 6229 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccaa7f752f254c5b9200fb381663e011 in term 1.
I20260924 07:20:47.759984 6154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccaa7f752f254c5b9200fb381663e011" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "386f9e2791834b66816bf885644a20e0"
I20260924 07:20:47.760171 6154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:20:47.757653 6283 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.760586 6283 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:47.761507 6154 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccaa7f752f254c5b9200fb381663e011 in term 1.
W20260924 07:20:47.762312 6058 tablet.cc:2369] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:20:47.762746 6056 leader_election.cc:302] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0aab15eaa1bb4329bdfe9e2d0b0abc04, ccaa7f752f254c5b9200fb381663e011; no voters:
I20260924 07:20:47.766577 6283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:20:47.766781 6283 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [term 1 LEADER]: Becoming Leader. State: Replica: ccaa7f752f254c5b9200fb381663e011, State: Running, Role: LEADER
I20260924 07:20:47.767095 6283 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } }
I20260924 07:20:47.771025 6186 master_runner.cc:263] No leader master found from master 0aab15eaa1bb4329bdfe9e2d0b0abc04
I20260924 07:20:47.771088 6186 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:47.771919 6037 master_runner.cc:263] No leader master found from master ccaa7f752f254c5b9200fb381663e011
I20260924 07:20:47.771992 6037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:20:47.773411 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.760959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.15.127.188:44581)
0924 07:20:47.761096 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.761100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.763408 (+ 2308us) server_negotiation.cc:407] Connection header received
0924 07:20:47.763567 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.763571 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.763623 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.763697 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.764735 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.765607 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.767090 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.767459 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.767543 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.769509 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.769513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.769516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.769542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.771450 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.771940 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.771944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.771948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.772044 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.772355 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.772358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.772360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.772624 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.772732 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.773121 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.773244 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:20:47.773629 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.761678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.15.127.190:33437)
0924 07:20:47.761828 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.761832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.763749 (+ 1917us) server_negotiation.cc:407] Connection header received
0924 07:20:47.763920 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.763924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.763969 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.764020 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.764908 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.765659 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.767106 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.767443 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.767533 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.769544 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.769548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.769550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.769574 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.771784 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.772265 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.772268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.772273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.772354 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.772693 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.772696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.772698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.772907 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.773029 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.773361 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.773484 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 07:20:47.774134 6310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaa7f752f254c5b9200fb381663e011. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
W20260924 07:20:47.774188 6277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 07:20:47.774266 6310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: This master's current role is: LEADER
W20260924 07:20:47.774353 6277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:20:47.776048 6312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:20:47.779356 6312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:20:47.783015 6185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:47.762873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.15.127.189:38255)
0924 07:20:47.763109 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.763113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.773929 (+ 10816us) server_negotiation.cc:407] Connection header received
0924 07:20:47.774530 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.774536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.774596 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.774649 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.775494 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.776237 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.777296 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:47.777603 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:47.777682 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:47.779377 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:20:47.779381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:20:47.779384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:20:47.779412 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:20:47.781098 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.781597 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.781604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.781606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.781693 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:20:47.782035 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:20:47.782038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:20:47.782040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:20:47.782249 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:20:47.782347 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:47.782699 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 07:20:47.782841 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260924 07:20:47.790076 6229 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [term 1 FOLLOWER]: Refusing update from remote peer ccaa7f752f254c5b9200fb381663e011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:47.790787 6283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:47.795487 6266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaa7f752f254c5b9200fb381663e011. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.795639 6266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:47.795701 6154 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [term 1 FOLLOWER]: Refusing update from remote peer ccaa7f752f254c5b9200fb381663e011: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:20:47.796316 6283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [LEADER]: Connected to new peer: Peer: permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:20:47.800048 6265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccaa7f752f254c5b9200fb381663e011. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.800166 6265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:47.807461 6310 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.807597 6310 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:47.808679 6266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.808809 6266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:47.809034 6314 mvcc.cc:205] Tried to move back new op lower bound from 7332800298140188672 to 7332800298057146368. Current Snapshot: MvccSnapshot[applied={T|T < 7332800298140188672}]
I20260924 07:20:47.809185 6315 mvcc.cc:205] Tried to move back new op lower bound from 7332800298140188672 to 7332800298057146368. Current Snapshot: MvccSnapshot[applied={T|T < 7332800298140188672}]
I20260924 07:20:47.811203 6283 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.811335 6283 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011 [sys.catalog]: This master's current role is: LEADER
I20260924 07:20:47.812317 6265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccaa7f752f254c5b9200fb381663e011" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccaa7f752f254c5b9200fb381663e011" member_type: VOTER last_known_addr { host: "127.15.127.190" port: 33437 } } peers { permanent_uuid: "386f9e2791834b66816bf885644a20e0" member_type: VOTER last_known_addr { host: "127.15.127.189" port: 38255 } } peers { permanent_uuid: "0aab15eaa1bb4329bdfe9e2d0b0abc04" member_type: VOTER last_known_addr { host: "127.15.127.188" port: 44581 } } }
I20260924 07:20:47.812433 6265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:20:47.812904 6317 mvcc.cc:205] Tried to move back new op lower bound from 7332800298140188672 to 7332800298057146368. Current Snapshot: MvccSnapshot[applied={T|T < 7332800298140188672}]
I20260924 07:20:47.814803 6312 catalog_manager.cc:1347] Generated new cluster ID: 8913e1fab41042a7a61fc79a6f2d9cc2
I20260924 07:20:47.814867 6312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:20:47.840229 6312 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:20:47.841648 6312 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:20:47.854074 6312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccaa7f752f254c5b9200fb381663e011: Generated new TSK 0
I20260924 07:20:47.854970 6312 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:20:47.885555 6261 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.883068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.15.127.188:44581)
0924 07:20:47.883195 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.883199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.884442 (+ 1243us) server_negotiation.cc:407] Connection header received
0924 07:20:47.884634 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.884637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.884683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.884734 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.885414 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58900: BlockingRecv error: recv got EOF from 127.0.0.1:58900 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 07:20:47.887280 6108 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.883827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.15.127.190:33437)
0924 07:20:47.883966 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.883970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.886262 (+ 2292us) server_negotiation.cc:407] Connection header received
0924 07:20:47.886489 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.886492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.886542 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.886591 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.887160 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58174: BlockingRecv error: recv got EOF from 127.0.0.1:58174 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 07:20:47.889353 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.888012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51598 (local address 127.15.127.189:38255)
0924 07:20:47.888123 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.888127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.888468 (+ 341us) server_negotiation.cc:407] Connection header received
0924 07:20:47.888636 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.888639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.888686 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.888735 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.889241 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51598: BlockingRecv error: recv got EOF from 127.0.0.1:51598 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 07:20:47.893585 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.892165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51610 (local address 127.15.127.189:38255)
0924 07:20:47.892250 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.892254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.892676 (+ 422us) server_negotiation.cc:407] Connection header received
0924 07:20:47.892873 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.892877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.892962 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.893034 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.893498 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51610: BlockingRecv error: recv got EOF from 127.0.0.1:51610 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 07:20:47.898502 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.897123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.15.127.189:38255)
0924 07:20:47.897210 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.897214 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.897641 (+ 427us) server_negotiation.cc:407] Connection header received
0924 07:20:47.897830 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.897833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.897892 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.897938 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.898416 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51618: BlockingRecv error: recv got EOF from 127.0.0.1:51618 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 07:20:47.904137 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.902674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.15.127.189:38255)
0924 07:20:47.902770 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.902774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.903292 (+ 518us) server_negotiation.cc:407] Connection header received
0924 07:20:47.903480 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.903485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.903531 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.903583 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.904048 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51632: BlockingRecv error: recv got EOF from 127.0.0.1:51632 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 07:20:47.911906 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.910428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.15.127.189:38255)
0924 07:20:47.910553 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.910557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.911059 (+ 502us) server_negotiation.cc:407] Connection header received
0924 07:20:47.911258 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.911261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.911309 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.911355 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.911815 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51642: BlockingRecv error: recv got EOF from 127.0.0.1:51642 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 07:20:47.924593 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.922904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.15.127.189:38255)
0924 07:20:47.923051 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.923056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.923573 (+ 517us) server_negotiation.cc:407] Connection header received
0924 07:20:47.923783 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.923786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.923842 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.923901 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.924485 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51646: BlockingRecv error: recv got EOF from 127.0.0.1:51646 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 07:20:47.945297 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.943407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51650 (local address 127.15.127.189:38255)
0924 07:20:47.943533 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.943538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.944143 (+ 605us) server_negotiation.cc:407] Connection header received
0924 07:20:47.944364 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.944368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.944427 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.944489 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.945147 (+ 658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51650: BlockingRecv error: recv got EOF from 127.0.0.1:51650 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 07:20:47.981197 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:47.980025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.15.127.189:38255)
0924 07:20:47.980174 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:47.980179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:47.980334 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:20:47.980460 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:47.980465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:47.980521 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:47.980623 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:47.981068 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51660: BlockingRecv error: recv got EOF from 127.0.0.1:51660 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 07:20:48.048488 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:48.047169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.15.127.189:38255)
0924 07:20:48.047328 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.047350 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.047514 (+ 164us) server_negotiation.cc:407] Connection header received
0924 07:20:48.047698 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.047700 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.047748 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.047847 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:48.048335 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51664: BlockingRecv error: recv got EOF from 127.0.0.1:51664 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 07:20:48.180001 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:48.178743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.15.127.189:38255)
0924 07:20:48.178899 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.178904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.179050 (+ 146us) server_negotiation.cc:407] Connection header received
0924 07:20:48.179232 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.179236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.179283 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.179374 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:48.179882 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51666: BlockingRecv error: recv got EOF from 127.0.0.1:51666 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 07:20:48.439265 6185 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:20:48.438075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51676 (local address 127.15.127.189:38255)
0924 07:20:48.438212 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.438219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.438319 (+ 100us) server_negotiation.cc:407] Connection header received
0924 07:20:48.438500 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.438509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.438576 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.438669 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:20:48.439129 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51676: BlockingRecv error: recv got EOF from 127.0.0.1:51676 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 07:20:48.758556 6289 catalog_manager.cc:1259] Loaded cluster ID: 8913e1fab41042a7a61fc79a6f2d9cc2
I20260924 07:20:48.758656 6289 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: loading cluster ID for follower catalog manager: success
I20260924 07:20:48.760360 6289 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: acquiring CA information for follower catalog manager: success
I20260924 07:20:48.761735 6289 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0aab15eaa1bb4329bdfe9e2d0b0abc04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:20:48.775599 6277 catalog_manager.cc:1259] Loaded cluster ID: 8913e1fab41042a7a61fc79a6f2d9cc2
I20260924 07:20:48.775707 6277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: loading cluster ID for follower catalog manager: success
I20260924 07:20:48.777374 6277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: acquiring CA information for follower catalog manager: success
I20260924 07:20:48.778481 6277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 386f9e2791834b66816bf885644a20e0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:20:48.957924 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.953542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.15.127.189:38255)
0924 07:20:48.953796 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.953801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.953899 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:20:48.954092 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.954095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.954226 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.954341 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:48.954624 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.955226 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.956279 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.956515 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.956630 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.956969 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:48.957504 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:48.957642 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.957705 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.957761 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
I20260924 07:20:48.964329 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.960951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.15.127.188:44581)
0924 07:20:48.961215 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.961218 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.961231 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:20:48.961341 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.961344 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.961456 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.961598 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:48.961756 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.962288 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.963072 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.963284 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.963358 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.963601 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:48.964027 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:48.964116 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.964168 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.964213 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
I20260924 07:20:48.971278 6385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.967183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.15.127.189:38255)
0924 07:20:48.967536 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.967540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.967555 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:20:48.967655 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.967660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.967714 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.967796 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:48.968124 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.968800 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.970071 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.970358 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.970442 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.970745 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:48.970820 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:48.970888 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.971059 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.971103 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278}
I20260924 07:20:48.971727 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.967316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58190 (local address 127.15.127.190:33437)
0924 07:20:48.967530 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.967536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.967940 (+ 404us) server_negotiation.cc:407] Connection header received
0924 07:20:48.968040 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.968047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.968206 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.968310 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:48.968659 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.969317 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.970250 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.970540 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.970660 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.970963 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:48.971297 (+ 334us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:48.971401 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.971493 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.971556 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":68,"threads_started":1}
I20260924 07:20:48.972193 6391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:20:48.967192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.15.127.188:44581)
0924 07:20:48.967296 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:20:48.967299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:20:48.967801 (+ 502us) server_negotiation.cc:407] Connection header received
0924 07:20:48.967953 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:20:48.967956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:20:48.968001 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:20:48.968075 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:20:48.968304 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.968912 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.970499 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:20:48.970778 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:20:48.970886 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:20:48.971721 (+ 835us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:20:48.971791 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:20:48.971834 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:20:48.972020 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 07:20:48.972063 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:20:48.974982 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 6037
I20260924 07:20:48.981840 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 6112
I20260924 07:20:48.987840 15870 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYFHG9/build/debug/bin/kudu with pid 6186
2026-09-24T07:20:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3048 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6077 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285339 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST